Cisco WLC and OpenSSL Madness

I was trying to convert a PFX file to be used for webauth on a Cisco WLC running the latest 7.x code. I've done this several times before in past lives and expected no resistance, but I could not get the WLC to accept the PEM certificate converted by OpenSSL 1.0 from a CentOS 6.3 server.

The WLC kept reporting a "private key decode failed".

However, one stroke of luck I found this post: https://supportforums.cisco.com/thread/340194

The forum member recommended using OpenSSL for windows at a version <1.0. One attempt and it went beautifully, the WLC accepted the certificate!

Does anyone have any ideas? Hopefully this helps someone else.

IPv6 Application Support in a state of Flux

Lots of people go on and on about IPv6. Some heralding its arrival (see World IPv6 Launch Day) and some gnashing their teeth (every server admin out there who loves doing everything by address). I'm not going to get into that. Its coming, and it will be a great thing once it gets here, but he growing pains are going to hurt.

I'm going to talk through two recent headaches caused by having IPv6 enabled.

First, Windows Server 2008 R2. IPv6 is enabled by default, and when recently building a new domain controller, having IPv6 on (and more importantly, what it does by default) gave me a world of hurt.

By default, the IPv6 stack is preferred over IPv4. This means that for any outbound connection, if the system thinks it can use IPv6, it will (try to) do so. Additionally, by default, the IPv6 stack uses the ::1 IPv6 loopback address as its default nameserver on my Windows 2008 R2 server. (I can't find any documentation as to why or if this is normal)

Microsoft themselves say never to configure a DC running the DNS role to use itself as the first/primary DNS server, it can lead to a race condition of sorts.

So the combination of these two default behaviors gave me a killer headache when adding the AD DS and DNS roles to a 2008 R2 server. The AD DS was unable to replicate the AD databases, including the DNS zones, due to failed DNS lookups against an empty server. I was only able to find this by nslookup defaulting to ::1 as the nameserver to search against when I was troubleshooting.

Secondly, like Windows 2008 and Windows 7, Java 7 also prefers IPv6 over IPv4. However, poor little Java has no indication service to know if the IPv6 stack is functioning properly or not. (Windows 7/2008, outside of Java, does this quite well, Java should pay attention to how MS tracks it). So if you are having issues (like me) running Java Applications or launching Web Starts, this blog points out the ever-so-graceful solution in Windows. You can also try adding these to the startup arguements to the java command line calling your jnlp or jar files, but I don't recommend it for Java Web Start applications, they are usually meant to be launched on-demand from a browser.

-J-Djava.net.preferIPv4Stack=true

PS, if you are running AnyConnect like the blogger was, you must sign out of your current session and reconnect before the change is picked up.

IGMP Snooping-Enabled NLB on Cisco IOS

Microsoft NLB. What can I say...its free, and its Microsoft, you're not getting a premo solution. In a virtual environment, where a NLB member can move from physical server to physical server, some real fun begins. Many HOWTOs, including Cisco's own, will have you placing static CAM entries everywhere. BLEH! I hope to show you how to avoid that.

The MS rundown of NLB as a whole:
http://technet.microsoft.com/en-us/library/bb742455.aspx

Cisco's HOWTO (needs work as you'll see below):
http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml

There's a problem with this HOWTO from Cisco, its a bit messy. I want to give some credit first, I gathered some of this data from this forum post, I wouldn't have pieced it together otherwise.

Lets begin.

Microsoft NLB, when running in IGMP-enabled multicast mode(at least in 2008 R2), uses a IANA multicast MAC address, not a non-IANA one. This is an important point that I think has been overlooked by Cisco in their guide...because with this, you don't need the static CAM entry, you just need IGMP Snooping.

IGMP snooping won't work without IGMP joins being seen from the servers (virtual or otherwise). So you need a IGMP router on that VLAN/segment to advertise its presence so the Windows servers respond, and the snooping is performed. To do that, you need to either A) Enable PIM (and therefore IGMP) on the interface or B) Simply enable the interface to be a "IGMP Querier". I'll leave it up to the reader to find their own platform's Multicast configuration guide to find the commands, but I will warn you of two things:

1) Make sure multicast-routing is turned on(in the VRF your interface is on if you're doing VRFs)
2) You will not see any "joined groups" in your show ip igmp command output

Finally, Cisco's note about process switching. Bug CSCsw87563 addresses it for the 6500 platform, not sure about the others. In my environment, I've added zero CAM entries because the bug is "fixed" for my platform, if you're in the same boat, good for you. If not...you really should, process switching is terrible. Even if you are affected by this, you only need to put the static cam on the switch with the SVI. All downstream switches will snoop and L2 switch with ease.

>>>>>A quick bug toolkit search revealed nothing on the popular 3560/3750 and 4500 lines. I am very interested if anyone can find more info this process switch thing on other platforms....even NX-OS!

Finally, all my work has been around avoiding the use of tying a static CAM entry to a physical interface everywhere (to avoid switch flooding). You still need a single static ARP entry.