Cisco 6880X & 6800ia Part 3

We continue to have problems with these boxes, here is the latest 'Bus Error":

This was in production so we took about a 15 minute downtime, not pretty. Still working with Cisco TAC to find the root cause.

For Palo Alto Networks Cult members :)

https://www.nsslabs.com/blog/seriously

Floor is open for comments :)

Cisco 6880X & 6800ia Part 2

Our deployment of 6880’s and 6800ia’s is in a large healthcare system, which in hindsight was not the best move. We received the 6880’s with 15.1(2)SY1 code which had some serious issues, the biggest being SDP error messages which was causing 6800 extenders to flap constantly. Upon contacting Cisco TAC, I was told to upgrade the code to 15.1(2)SY2 (TAC engineer knew about it beforehand but there was no official documentation available….hmmm :)

Upgrading to SY2 did fix the flapping issue but caused a few other problems. Most important was that the extenders were getting stuck during the code upgrade (while pulling the new code from the parent switch). After a month long correspondence with TAC and BU engineers, we were told that “most” of these issues have been addressed in SY3 release of the code so we should upgrade to SY3. Keep in mind that every time you upgrade the 6880, the attached fex’s have to pull down the new code all over again and in the case of VSS, the fex’s reboot twice before during the process (when using eFSU). This may not be a big deal for a small setup but it is a huge problem in a 24x7 hospital environment. This problem is compounded by the fact that fex’s take 6-9 minutes to be fully operational after the reload. To make a long story short, even after the upgrade to SY3 we are still having major issues (including VSL link failure).

Here is a list of outstanding issues for 6880x (August 31, 2014):

• Random VSL Link Failure.
•  “ENTROPY_FAILURE: Unable to collect sufficient entropy”
•  SSH stops working if the active switch goes into recovery mode. The only “fix” we have found so far is to reload both shelves.
• 7-9 minutes boot time for the fex’s.
•  In case of a single homed fex, uplink interface shutdown/failure or a simple twinax/fiber failure will reboot the fex (per cisco, it’s a “security” feature but it’s an issue for me).
• ISSU/eFSU doesn’t provide much visibility into the upgrade process, leaving you wondering if it is stuck in the process (per cisco SY3 will show more “messages” during the upgrade process, I haven’t seen anything new so far).

We have received 12 of these boxes and have had 3 DOA linecards (C-6880-X-16P10G) so far. Keep in mind that you can’t interchange C-6880-X-LE-16P10G & C-6880-X-16P10G linecards as the LE is for Lite Edition (smaller hardware table) and will absolutely NOT work on the X (bigger hardware table) chassis. Installing/inserting these linecards into the chassis is tricky as well. If you don’t have it aligned exactly at the proper angle, it will get stuck and you will not be able to yank it out without messing up something else…..poorly designed linecard to say the least.

If you are not bothered by any of this stuff then you are a good candidate for deploying 6880’s J

Cisco 6880X & 6800ia Part 1

If you are thinking about deploying 6880X with some 6800ia’s then you may want to read this post in its entirety before making the final decision. I will be updating either this post or adding new posts as I come across new/relevant information.

I’ve long been a Cisco fan and one could have sold me a brick with a cisco logo on it (up until a couple of years ago) and I would have been very happy with that purchase. I would still have the same love for Cisco had I not touched the Cisco ACS (4.x & 5.x), Cisco Prime LMS 4.1 in the last few years, and the new 6880X with 6800 instant access switches. For now, let’s forget about the ACS/LMS and discuss 6880/6800ia. Being that routing and switching is one of Cisco’s core competencies; one would expect a very stable, reliable, and a feature rich (new features) product (like the 6500, Nexus 7K, ASR etc.). Unfortunately 6880X doesn’t enjoy any of those traits for now.

There is no doubt that the idea is good, you take the same model as nexus 7K/5K with fex’s and make it available outside of the datacenter environment but the execution of this plan has been subpar. With that being said, let us start with some of the “good” stuff about these boxes:

    1.      Extremely competitive price (compared to the 6500/6807 with Sup 2t).
    2.      PoE availability on the 6800 instant access switches.
    3.      Great 10g port density for the price.
    4.      Feature rich (L2/L3, full MPLS, GRE in hardware).

Please note that my assumption of “good” is heavily based on the pricing.

6880X datasheet:

http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6880-x-switch/data_sheet_c78-728228.html

And now some of the “not so good” list:

1.  Max instant access switch/fex ports restricted to 1008. This means that you can only deploy 21 6800ia’s switches/fex’s per VSS pair. Per Cisco, this number will be increased to perhaps 2000 ports or more by the end of the year.
2. You can only stack up to (3) 6800ia switches.
3. You can only use FEX id’s 1-12 for now. So if you have a deployment where you need 15 single 6800ia switches….well you can only deploy 12 for now J 

Here is a complete list of restrictions and the things you can’t do on the 6800ia’s:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/15-1SY/config_guide/sup2T/15_1_sy_swcg_2T/instant_access.pdf

In the next post, I will share our 6880/6800ia deployment and the ongoing struggles with these boxes.

CCIE R/S V5

It's finally official:
http://www.cisco.com/web/learning/certifications/expert/ccie_rs/index.html

Some of the major items removed from the lab:
1. Frame Relay
2. PfR
3. Layer 2 QoS
4. Q-in-Q tunneling
5. IOS Firewall / ZBF

Major additions:
1. DMVPN
2. IPsec

Overall, the change is not too drastic.

Happy Hunting :)

CCIE # 40567

Finally got my ccie #. No, I didn't buy it off of ebay, I actually passed the R/S lab :)  It was a very humbling experience to say the least, even for someone like me who has been through this ordeal before.  My last failed attempt was in 2008 and at the time I thought I was closing the ccie book for “good” as it was very demoralizing to have failed multiple times. Well, all I can say now is that it was a big mistake on my part to give up when I did as the lab was a lot easier at that time compared to what it is now. In any case, late last year one of my friends who works for Cisco started talking about taking the lab and kept telling me that I need to give it another shot as well. I didn’t really pay much attention to him until he called me one day in March and told me that he passed his ccie lab. Not sure what came over me but I decided that I will give it another shot. I went all out and studied as much as I possibly could with heavy emphasis on the troubleshooting section.

I took the lab on a Saturday as that was the last day before my written exam would expire. The proctor was pretty cool but a bit hyper. He lined us up and gave an almost 15 minute lecture about the lab rules and then let us loose. I jumped right into the troubleshooting section (nervous as heck)  and after two hours I thought I had lost the troubleshooting battle as I was unable to resolve a 3 point ticket and doubtful about another  2 point ticket. Going into the configuration section I was a lot calmer but still disappointed because I wasn’t sure if I passed the troubleshooting section. I finished the lab about an hour and fifteen minutes ahead of time and only spent about 10 minutes verifying. I left thinking that I failed the lab yet again. The flight from RTP back to Houston had to be one of the longest and the most uncomfortable flight I have had. As soon as I landed, I checked my email and saw that the score the lab score is available. I was so depressed that I didn’t even bother checking the score and just drove back home from the airport. It was almost midnight when I got home and walked in quietly as I didn’t want to wake up the little ones. I apologized to my wife for taking all the time away from the family and studying but still not passing. She told me not to worry about it and to just relax and perhaps I will get it the next time around. About twenty minutes after I got home, my wife goes, “well, why don’t you check your score to see if you at least did better than the last time”. I asked her to login instead as I didn’t have the heart to see myself, she logs in and after a few seconds she goes, “oh I think you passed” and I’m thinking she is probably looking at my written exam score. And then she goes on to read my CCIE number, I felt as if I was dreaming but when I heard her screaming and the kids woke up, I realized that it was for real. I checked it again and again and a few more times. My wife even went as far as taking the pictures of the screen just in case something gets changed. I was so excited that I couldn’t sleep but finally had to as I was extremely exhausted. When I woke up, I logged in and verified yet again to make sure that my ccie # didn’t disappear overnight…..it was still there.  

One thing I have noticed is that I have more pressure at work now because everyone just expects me to know everything there is to know because I’m a CCIE. It’s a good thing that others look up to you but it can also be a pain because you have to be able to pick up new technologies on the fly and provide solutions. For now, I’m just taking it easy and enjoying my number, but I know that pretty soon I will be bored and perhaps start thinking about another ie….datacenter…maybe :)

Cisco Nexus - vPC

Here are few things right out of cisco documentation to keep in mind when deploying vPC's. I couldn't find all this at one place in the Cisco documentation (maybe I didn't look hard enough:) so I decided to put it here for those needing a quick fact check:

1. vPC peer link MUST consist of 10G ports.
2. vPC domains can't be stretched across multiple VDCs on the same switch.
3. vPC can't contain links that are terminated on different VDCs on the same switch.
4. You can only have ONE vPC domain per VDC or switch.
5. vPC domain can't have more than two peer switches or VDCs.
6. VDC configured for vPC must have its own vPC keepalive link and vPC peer link.
7. Since vPC is a layer 2 port channel, vPC doesn't support L3 port channels.