IGMP Snooping-Enabled NLB on Cisco IOS

Microsoft NLB. What can I say...its free, and its Microsoft, you're not getting a premo solution. In a virtual environment, where a NLB member can move from physical server to physical server, some real fun begins. Many HOWTOs, including Cisco's own, will have you placing static CAM entries everywhere. BLEH! I hope to show you how to avoid that.

The MS rundown of NLB as a whole:
http://technet.microsoft.com/en-us/library/bb742455.aspx

Cisco's HOWTO (needs work as you'll see below):
http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml

There's a problem with this HOWTO from Cisco, its a bit messy. I want to give some credit first, I gathered some of this data from this forum post, I wouldn't have pieced it together otherwise.

Lets begin.

Microsoft NLB, when running in IGMP-enabled multicast mode(at least in 2008 R2), uses a IANA multicast MAC address, not a non-IANA one. This is an important point that I think has been overlooked by Cisco in their guide...because with this, you don't need the static CAM entry, you just need IGMP Snooping.

IGMP snooping won't work without IGMP joins being seen from the servers (virtual or otherwise). So you need a IGMP router on that VLAN/segment to advertise its presence so the Windows servers respond, and the snooping is performed. To do that, you need to either A) Enable PIM (and therefore IGMP) on the interface or B) Simply enable the interface to be a "IGMP Querier". I'll leave it up to the reader to find their own platform's Multicast configuration guide to find the commands, but I will warn you of two things:

1) Make sure multicast-routing is turned on(in the VRF your interface is on if you're doing VRFs)
2) You will not see any "joined groups" in your show ip igmp command output

Finally, Cisco's note about process switching. Bug CSCsw87563 addresses it for the 6500 platform, not sure about the others. In my environment, I've added zero CAM entries because the bug is "fixed" for my platform, if you're in the same boat, good for you. If not...you really should, process switching is terrible. Even if you are affected by this, you only need to put the static cam on the switch with the SVI. All downstream switches will snoop and L2 switch with ease.

>>>>>A quick bug toolkit search revealed nothing on the popular 3560/3750 and 4500 lines. I am very interested if anyone can find more info this process switch thing on other platforms....even NX-OS!

Finally, all my work has been around avoiding the use of tying a static CAM entry to a physical interface everywhere (to avoid switch flooding). You still need a single static ARP entry.

Infoblox API Scripting

Infoblox makes a pretty sweet little appliance, providing DNS(ISC BIND), DHCP(ISC DHCPD), TFTP/HTTP File distribution for your enterprise. More or less its a Linux appliance with a decent GUI on top of it for the aforementioned features.

One of my favorite features about it is its API, and I wanted to share some of my experience with it. Its entirely Perl based, and getting it setup was painful with me with CPAN, but their binary package on my CentOS box worked a treat. While (in my opinion) the API's documentation isn't the best, it has some very vague descriptions of many of the functions, and the examples they give aren't much for mass-modification purposes, but for creating new networks/ranges.

I'm not going to do a starter guide, you'll have to read their docs for that....I'll just provide some of my coding to supplement their existing documentation.

First of all, check out "ibcli", I used it as MY supplement to figure out the right data structure/method to use when writing this script.

The purpose of my script was to help facilitate moving from a single Infoblox HA pair to a failover set of geographically separated "Grid Member" HA pairs for even more DHCP fault-tolerance for our WAN users in case of a large network outage.

When moving to this setup, you must reconfigure every network and every DHCP range to "point" to the "failover set" consisting of a pairing between the two failover Infoblox Grid Members.
Since we have several hundred, DHCP Networks, my script was designed to dump to STDOUT all Networks and Ranges on the appliance before the change, then change all Networks/Ranges (save a few special setups I exempted) to point to the failover set, and then dump the "post operation" configuration to STDOUT for verification.

Please keep in mind that this script could definitely use a lot more catch statements for error handling, but I kept this pretty lean just to do this one job for me.

Without further adieu, see attached. Rename it from the .txt extension to .pl.

IBConvert Script

Adding a Cisco 3750 Switch to an Existing Stack

Another simple task that Cisco doesn't quite make 100% clear.

Let's say you have an existing Cisco 3750 Switch Stack and you have had a bit of an office redesign and you now need to add another member to the stack to add additional port capacity for the new users moving in.

This is mostly simple to do, but there are several checks to make along your journey, and hopefully I can point them out clearly.

1. First, check the stacking status of your existing stack using the below command. It should say that the stack ring speed is "full". If it doesn't you need to ensure you have a complete stack "ring" thereby having redundant stacking paths on each switch.
   
   

   ZZStack#show switch stack-ring speed
   
   Stack Ring Speed        : 32G
   Stack Ring Configuration: Full
   Stack Ring Protocol     : StackWise

   
   
   See this publication for stack-wiring help:
   http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/hardware/quick/guide/3750GSG3.html#wp49279
   
   
2. Once you know your stack is a complete ring("full") you can safely break this ring to insert your new switch. ENSURE YOUR NEW SWITCH IS POWERED OFF. Rack this new switch adjacent to your existing switches. "Break" the ring in one place so you can wire your new switch so it fits nice and neat in this new stack, following the wiring schema linked above. Once everything is wiring in place, you may NOW power on your new switch.
   
   
3. After a few minutes your switch will have booted and the existing stack should recognize it as joining the stack. You can verify its status with:
   
   

   ZZStack#show switch stack-ring speed
   
   Stack Ring Speed        : 32G
   Stack Ring Configuration: Full
   Stack Ring Protocol     : StackWise

   
   
   

   ZZStack#show switch
   Switch/Stack Mac Address : 0024.9803.8e80
                                              H/W   Current
   Switch#  Role   Mac Address     Priority Version  State
   ----------------------------------------------------------
   *1       Master 0024.9803.8e80     15     0       Ready
    2       Member 0023.ac0f.7880     1      0       Ready
    3       Member 001a.e267.0080     1      0       Version Mismatch

   
   
   In this example switch 3 was added to the stack, the stack ring looks good, but on the switch status output, instead of saying "Ready" it says "Version Mismatch." You now need to ensure your switch gets the same IOS version as its stack-mates and reboots for this to take effect(see below).
   
   If your stack says your new switch is in the "Ready" state, you are in luck and you are done! (except for configuring your new user ports)
   
   
4. Automatic Upgrade is a great thing in theory, but I've not had the best of luck with it. From what I can tell if your existing stack and this new member you are adding are running different IOS featuresets, the automatic upgrade will NOT WORK.
   
   To check to see if its working or not, check the log. If you see log entries resembling any of the below, it appears automatic upgrade is working as it should.
   

   ZZStack#show log
   ---snip---
   Nov 24 16:58:36.388 EST: %IMAGEMGR-6-AUTO_COPY_SW_INITIATED: Auto-copy-software process initiated for switch number(s) 3

   
   This shows that it has started the auto upgrade process. You can check the status with the "show archive status" command or by continually checking the log.
   
   If you see the below it was successful and is now rebooting this member switch so the correct IOS loads and it can finally join the stack as it should(and you should be done):
   

   ---snip---
   Nov 24 17:06:04.453 EST: %IMAGEMGR-6-AUTO_COPY_SW: Software successfully copied to
   Nov 24 17:06:04.453 EST: %IMAGEMGR-6-AUTO_COPY_SW: system(s) 3
   Nov 24 17:06:04.453 EST: %IMAGEMGR-6-AUTO_COPY_SW: Done copying software
   Nov 24 17:06:04.453 EST: %IMAGEMGR-6-AUTO_COPY_SW: Reloading system(s) 3

   
   
   If you don't see any of the above log entries, go on to the next step.
   
   
5. This is the step they don't really explain well. Thankfully, the "Version Mismatch" state, while not activating any of its ports, does allow you to manipulate the flash filesystem of the inconsistent member so you can stage it from the main stack interface.
   
   If you made it to this step, it probably means there is a featureset mismatch(or some other problem) and you need to force this new member to take the IOS version that the stack is currently running. In this scenario, even the "archive copy-sw" command does not work, so you must either load the IOS bin file manually or use use the appropriate "archive download-sw" command with the "/allow-feature-upgrade" switch to load the IOS to the ENTIRE STACK again, including this new member(but you only need to reboot the new member). I prefer to use the archive command, its slow but its so easy!
   
   Here's the link to the software upgrade caveats/howtos for stack configurations(if it seems like I breezed through this last step):
   http://www.cisco.com/en/US/products/hw/switches/ps5023/products_configuration_example09186a00804799d7.shtml

Cisco Passive Interface Address-Family Warning

When you are configuring passive interfaces for the global VRF as well as other configured address-families, there is a difference between how the 12.4/12.4T Router IOS behaves and how a 6500/3750/3560 behaves. If you don't know what a passive interface is, this post isnt for you, but here is a step in the right direction.

On Routers, passive-interface settings are shared amongst ALL ADDRESS-FAMILIES, INCLUDING GLOBAL. While on the 3560/3750/6500 etc L3 Switch IOS, the address-families each have their own unique passive interface settings.

For instance:

router eigrp 1
 network 10.0.0.0
 no auto-summary
 passive-interface default
 !
 address-family ipv4 vrf VRF-A
  network 10.0.0.0
  no auto-summary
 exit-address-family
 !
 address-family ipv4 vrf VRF-B
  network 10.0.0.0
  no auto-summary
 exit-address-family
end

The above code snippet on a Cisco Router would cause no EIGRP neighbor relationships to form, in either VRF or the global route table. However, on a Cisco L3 switch, the passive-interface command would ONLY apply to the global route table and each VRF will not have any of its member interfaces be acting as passive.

This final code snippet is not valid on a Router IOS, but is perfectly valid(and desired) on a L3 Switch IOS, with Po1 in the global route table, Po2 in VRF-A, and Po3 in VRF-B:

router eigrp 1
 network 10.0.0.0
 no auto-summary
 passive-interface default
 no passive-interface Po1
 !
 address-family ipv4 vrf VRF-A
  network 10.0.0.0
  no auto-summary
  passive-interface default
  no passive-itnerface Po2
 exit-address-family
 !
 address-family ipv4 vrf VRF-B
  network 10.0.0.0
  no auto-summary
  passive-interface default
  no passive-interface Po3
 exit-address-family
end

Renumbering a Cisco 3750 Stack

For those of you familiar with the Cisco 3750 switches, their killer feature is their ability to "stack". Stacking sets one switch as the "master" and utilizing stack cables out of the back of the switch, each switch in the stack appears as one. This allows for expandability, and ease of management.

Anywho, one annoyance of stacking 3750s is that an engineer building or adding to a stack can often install switches "out of order" meaning their stack ID does not match their order as-racked. Basically, most English(or any other Western language) speaking peoples. would assume that a group of 5 switches in a rack would be numbered 1-5, top to bottom. If this isnt the case, ASSumptions can lead to some mistakes and mistakes lead to fussy customers. Yes, this can be checked by pressing the mode button on the front of any stack member, but isnt it better to not worry with that and have it right from the beginning?

Anyway, as a best practice, I always set the order as racked to match the stack order. Its not as simple as issuing the renumber subcommand as it won't move your switch port configurations with the move, you have to do some trickery to move the configurations(it even warns you of this when you hit the commands). If you want to renumber switches in a stack, here is a little HOWTO, since Cisco really doesn't go over this scenario, and I've done it a few times.

*************
Example:
*************
Stack of 2 switches in a rack, Switch 2 is on top, switch 1 is on the bottom.

Step 1: Prime the switches by setting their new stack numbers
We want 2 to become 1 and 2 to become 2, right?

In config mode:

switch 1 renumber 2
switch 2 renumber 1

Then write your config...simple enough.

Step 2:
Copy the startup config to your favorite text editor via screen scrape or file transfer. Do a find replace on all 2/0/ and replace with X/0/ and then find replace 1/0/ with 2/0/, and finally X/0/ with 1/0/. Basically you have flip-flopped all the port configs between the two switches using the power of find/replace.

Step 3:
Upload the file you edited above and have it overwrite the startup config of the stack. Reload the stack

Step 4:
Test. (of course).

Testing Copper Cables with Cisco Switches (TDR)

You can quickly detect any cable problems from your test with any 3560G/3750G/4500(with Gig Ports)/6500(with Gig Ports) by using the built-in IOS TDR. It will tell you the length of the cable and if there are any open pairs(and if so, where in the line they occur). This is better than running up to the location with a Fluke in hand to test cable problems.

First, you must run the test(its intrusive on the interface its being run on, so only run it on something that is OK to be unreachable for about a minute):

test cable-diagnostics tdr interface <InterfaceName>

After it completes, all you have to do is view the result:

show cable-diagnostics tdr interface <InterfaceName>

See the link below for the full scoop from Cisco:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.2_46_se/configuration/guide/swtrbl.html#wp1400280

How to Sort IP Addresses in Microsoft Excel

A strange combination of Excel tools will yield a method to sort a long list of IP Addresses. I'll start from copying a list from a Cisco Router ARP Table.

Here's how:

1. Paste the ARP Table into Excel, it will be a long line of text all in one column.


2. Select the column by clicking the letter at the top and run the "text to columns" tool.


3. Run through the short wizard using a space deliminator, should yeild you with 5 or so columns, one for IP, one for MAC, and the others.


4. Now, select the new column where the IP Addresses are, and run the same text to columns tool again, this time using a period as the deliminator.(you may want to first insert 3 blank columns to the right of the IP Address to make room for this new columnized data)


5. Now, sort by lowest to highest using first octet, then by second octet, then by third octet, and then by fourth octet.


6. Insert yet another column anywhere, and on line one use the CONCATENATE function as such(a1=first octet of first address, B1 as second of first, etc):



=CONCATENATE(A1,".",B1,".",C1,".",D1)



7. This should yield a cell with the address formatted properly. Select it and click the dot in the lower right hand corner of the selection and drag all the way down every row in your table, thereby copying the formula with relative cell modifications in your formula.


8. Now, select all these addresses in this column, and copy them.


9. By default, pasting them in another column will only paste formula data, so under Edit->Paste Special(or if you have 2007, click the arrow underneath the paste button) and select Paste Values.


10. Tada! You can now delete all 5 of you intermediary columns.

Native VLANs - ISL vs 802.1q, CatOS vs IOS

When configuring trunking, the discussion of Native VLANs may be a little confusing, especially when comparing ISL trunking vs 802.1q, and IOS vs CatOS.

Here's a breakdown:

First, the concept of Native VLAN is only understood by 802.1q. So, for ISL trunking, do not worry about setting it, as ISL encapsulates EVERY FRAME going through that trunk(more below).

A Native VLAN is merely a single VLAN ID already allowed on a trunk that will be used for all untagged frames sent/received. In other words, if the switch receives a frame on this trunk with no VLAN ID tagged on it, it will assume that this frame is in the Native VLAN, and inversely, if it sends a frame on this trunk that is in the Native VLAN, it will not bother tagging the frame.

As stated above, this does not apply to ISL because it encapsulates packets(instead of tagging them, like 802.1q), and every packet must be encapsulated, therefore, ISL does not bother with the concept.

Typically, you have a dedicated "switch management" VLAN on your network, and all your switches' management addresses are in this VLAN, and this VLAN is set as native on every trunk link between these switches. A lot of times setting the Native VLAN does not matter on newer switches, but I have found that some Cisco Autonomous APs and very old switches require it set, so I feel it is best to keep it consistent, YMMV.

To check the current Native VLAN on IOS:

sh int trunk InterfaceX/Y

To check the current Native VLAN on CatOS:

sh trunk X/Y

On IOS, its pretty easy to set the native VLAN, just enter the following command under the interface(where Z is the VLAN ID):

switchport trunk native vlan Z

On CatOS, its not as obvious, but just as easy, you just merely set the vlan of the trunk port as if it were an access port:

set vlan Z X/Y

As I said before its not terribly necessary nowadays, but for consistency I recommend setting it to the same VLAN as the management interface of your end-user switches. No matter what you do, you must ensure that the Native VLAN matches on both sides of your 802.1q trunk, because otherwise two Cisco switches will detect the mismatch via CDP and will block the port, until its fixed. The following IOS command will show you a list of any such blocked ports:

show spanning-tree inconsistentports

Cisco "Setup Express"

Newer Cisco "pizza box" switches (3560/3750 and others) have a MODE button on the front of them that is used for both the password-recovery feature, as well as the "setup express" feature.

Reference:
http://cciepursuit.wordpress.com/2007/06/29/completely-clearing-a-cisco-switchkinda/


I would recommend disabling the setup express feature(no setup express in configure mode) as it really provides NO benefit in my eyes. It does not affect the password recovery feature(unless you are super paranoid, disabling password recovery can be disastrous).

My argument is that while a switch is powered on, there should not be this easy to essentially reboot the switch with a blank config, with the only way to remedy is renaming files via console cable. This can have one nosy user/janitor take down your switch until you can run out there with a laptop.

PuTTY Connection Manager

PuTTY connection manager is a great idea in the works, but is needing a bit more development.

Basically, Its an app that "grabs" multiple PuTTY windows into one, tabbed interface. It supports having an encrypted proprietary database of all your connection entries, complete with macroing your login credentials post-login commands.

Its current version, 0.6.0, works well enough, but has several GUI bugs(window not coming into foreground when clicking into the PuTTY window), and its macros do not support any kind of EXPECT functionality(only perform next command if the previous one returns "x").

I still recommend it, beyond its annoyances, and hopefully they will be corrected soon. Here is a link to their site: http://puttycm.free.fr/

Nmap port scanning

As a network engineer, I find myself using nmap quite a bit. For those of you who don't know what nmap is, nmap.org is a great place to start finding out. :-) But basically, its a TCP/UDP port scanning tool. It works on windows and linux, and even has a pretty GUI(that I dont use...command line all the way!). They have some good tutorials on the site if you don't know how to use it, and in my opinion, every IT professional should know how to use it on a basic level.

Back to my original thought. Was doing some security audits here at work and someone reported that "icmp is open to server X from outside the network." I thought they meant icmp echo(ping). I found that echo is indeed blocked, so they must NOT have meant that. So I remembered that only about 10% of IT people really, actually know their stuff, so I went about trying to reproduce their results.

C:\>nmap -sO X.X.X.X

Starting Nmap 4.60 ( http://insecure.org ) at 2008-04-18 09:56 Eastern Daylight

Time

Interesting protocols on X.X.X.X:

Not shown: 255 open|filtered protocols

PROTOCOL STATE SERVICE

1        open  icmp

And there it was. Nmap reports that icmp is open. There are a bunch of types of icmp other than echo(but echo is by far the most popular ;) ) But why is it reporting icmp as open?

I'm not 100% sure why, but here is my hypothesis:

From http://nmap.org/book/man-port-scanning-techniques.html I know that when doing a UDP scan, the only way to know if a port is closed is if a ICMP Type 3 Code 3 is returned. So I know that if i did a UDP scan and any ports come back "closed" then technically, icmp is "open", since any said closed ports have been noted as closed due to a icmp message from this server.

However, this thought is now fubar'd since a UDP scan returned no closed ports....hmmm. Really, I'm kinda stumped. I need to find a good way to probe all ICMP messages to see whats supported. Cannot find anything on google initially...need to dig a bit more. (nmap does not seem to support icmp "scanning")