Wednesday, March 11, 2015

AIA-Enabled Certificate Chains

Have you ever wondered why your nice publicly-signed certificate displays just fine on your workstation but your mobile device shows a certificate error?

First, in review, all of your browsers trust an authority such as Verisign, so they have the Verisign Root in their trusted store. In the case of's certificate, its authority chain looks like so:

In order to fully trust the certificate, your browser must not only trust its built-in root (Baltimore CyberTrust Root), but the Intermediate certificate in-between. Your server should be configured to send all certificates in the chain to every HTTPS client.

If not presented by the server, most desktop browsers use AIA to dynamically download the Intermediate certificate, using the certificate's AIA field seen here:

If AIA works, no certificate error appears, because your desktop browser was able to complete the chain dynamically.

Mobile browsers, however, do not use the AIA extension. This is due to bandwidth/overhead of querying other websites to find the upstream Intermediate CAs. Desktops can handle this query process seamlessly, mobile browsers simply do not.

My recommendation is to send the full chain to everyone. This ensures no errors for mobile devices, and theoretically quicker load times because this AIA parsing does not need to be processed by any browsers. Inserting the Intermediate certificate in your webserver's CA configuration is the way to include this. Review your apache/IIS/other documentation on how to do this.

You can test your certificate chain with OpenSSL using the following command:
openssl s_client -showcerts -connect host.domain.tld:443

Everything is 100% right if you see the following at the bottom:
Verify return code: 0 (ok)

Friday, August 10, 2012

Cisco WLC and OpenSSL Madness

I was trying to convert a PFX file to be used for webauth on a Cisco WLC running the latest 7.x code. I've done this several times before in past lives and expected no resistance, but I could not get the WLC to accept the PEM certificate converted by OpenSSL 1.0 from a CentOS 6.3 server.

The WLC kept reporting a "private key decode failed".

However, one stroke of luck I found this post:

The forum member recommended using OpenSSL for windows at a version <1.0. One attempt and it went beautifully, the WLC accepted the certificate!

Does anyone have any ideas? Hopefully this helps someone else.

Thursday, August 2, 2012

IPv6 Application Support in a state of Flux

Lots of people go on and on about IPv6. Some heralding its arrival (see World IPv6 Launch Day) and some gnashing their teeth (every server admin out there who loves doing everything by address). I'm not going to get into that. Its coming, and it will be a great thing once it gets here, but he growing pains are going to hurt.

I'm going to talk through two recent headaches caused by having IPv6 enabled.

First, Windows Server 2008 R2. IPv6 is enabled by default, and when recently building a new domain controller, having IPv6 on (and more importantly, what it does by default) gave me a world of hurt.

By default, the IPv6 stack is preferred over IPv4. This means that for any outbound connection, if the system thinks it can use IPv6, it will (try to) do so. Additionally, by default, the IPv6 stack uses the ::1 IPv6 loopback address as its default nameserver on my Windows 2008 R2 server. (I can't find any documentation as to why or if this is normal)

Microsoft themselves say never to configure a DC running the DNS role to use itself as the first/primary DNS server, it can lead to a race condition of sorts.

So the combination of these two default behaviors gave me a killer headache when adding the AD DS and DNS roles to a 2008 R2 server. The AD DS was unable to replicate the AD databases, including the DNS zones, due to failed DNS lookups against an empty server. I was only able to find this by nslookup defaulting to ::1 as the nameserver to search against when I was troubleshooting.

Secondly, like Windows 2008 and Windows 7, Java 7 also prefers IPv6 over IPv4. However, poor little Java has no indication service to know if the IPv6 stack is functioning properly or not. (Windows 7/2008, outside of Java, does this quite well, Java should pay attention to how MS tracks it). So if you are having issues (like me) running Java Applications or launching Web Starts, this blog points out the ever-so-graceful solution in Windows. You can also try adding these to the startup arguements to the java command line calling your jnlp or jar files, but I don't recommend it for Java Web Start applications, they are usually meant to be launched on-demand from a browser.

PS, if you are running AnyConnect like the blogger was, you must sign out of your current session and reconnect before the change is picked up.

Thursday, February 9, 2012

IGMP Snooping-Enabled NLB on Cisco IOS

Microsoft NLB. What can I say...its free, and its Microsoft, you're not getting a premo solution. In a virtual environment, where a NLB member can move from physical server to physical server, some real fun begins. Many HOWTOs, including Cisco's own, will have you placing static CAM entries everywhere. BLEH! I hope to show you how to avoid that.

The MS rundown of NLB as a whole:

Cisco's HOWTO (needs work as you'll see below):

There's a problem with this HOWTO from Cisco, its a bit messy. I want to give some credit first, I gathered some of this data from this forum post, I wouldn't have pieced it together otherwise.

Lets begin.

Microsoft NLB, when running in IGMP-enabled multicast mode(at least in 2008 R2), uses a IANA multicast MAC address, not a non-IANA one. This is an important point that I think has been overlooked by Cisco in their guide...because with this, you don't need the static CAM entry, you just need IGMP Snooping.

IGMP snooping won't work without IGMP joins being seen from the servers (virtual or otherwise). So you need a IGMP router on that VLAN/segment to advertise its presence so the Windows servers respond, and the snooping is performed. To do that, you need to either A) Enable PIM (and therefore IGMP) on the interface or B) Simply enable the interface to be a "IGMP Querier". I'll leave it up to the reader to find their own platform's Multicast configuration guide to find the commands, but I will warn you of two things:

1) Make sure multicast-routing is turned on(in the VRF your interface is on if you're doing VRFs)
2) You will not see any "joined groups" in your show ip igmp command output

Finally, Cisco's note about process switching. Bug CSCsw87563 addresses it for the 6500 platform, not sure about the others. In my environment, I've added zero CAM entries because the bug is "fixed" for my platform, if you're in the same boat, good for you. If really should, process switching is terrible. Even if you are affected by this, you only need to put the static cam on the switch with the SVI. All downstream switches will snoop and L2 switch with ease.

>>>>>A quick bug toolkit search revealed nothing on the popular 3560/3750 and 4500 lines. I am very interested if anyone can find more info this process switch thing on other platforms....even NX-OS!

Finally, all my work has been around avoiding the use of tying a static CAM entry to a physical interface everywhere (to avoid switch flooding). You still need a single static ARP entry.

Monday, April 19, 2010

Disable NETBIOS with dhcpd

This was a difficult item to find...thanks to the dd-wrt wiki for the proper vendor string!!

NETBIOS is ancient...disabling NETBIOS altogether on networks(provided you are sure you want to do this) can be a good thing for both your network and your windows workstations.

For those of you using the default node-type, you will basically eliminate all netbios broadcasts on your network,and your workstations will ONLY use DNS for resolution, instead of DNS then WINS then NBT broadcasts.

Disadvantage is that you can no longer use tools like nbtstat to query machines on your network for their hostname (shouldn't you be using DDNS?) and connectivity using some 'short names' may be lost...but you should be using FQDNs whenever/wherever possible.

DHCP Option 43, value is "01:04:00:00:00:02"

The line in your ISC dhcpd config should be:

option vendor-encapsulated-options 01:04:00:00:00:02;

Tuesday, April 13, 2010

Infoblox API Scripting

Infoblox makes a pretty sweet little appliance, providing DNS(ISC BIND), DHCP(ISC DHCPD), TFTP/HTTP File distribution for your enterprise. More or less its a Linux appliance with a decent GUI on top of it for the aforementioned features.

One of my favorite features about it is its API, and I wanted to share some of my experience with it. Its entirely Perl based, and getting it setup was painful with me with CPAN, but their binary package on my CentOS box worked a treat. While (in my opinion) the API's documentation isn't the best, it has some very vague descriptions of many of the functions, and the examples they give aren't much for mass-modification purposes, but for creating new networks/ranges.

I'm not going to do a starter guide, you'll have to read their docs for that....I'll just provide some of my coding to supplement their existing documentation.

First of all, check out "ibcli", I used it as MY supplement to figure out the right data structure/method to use when writing this script.

The purpose of my script was to help facilitate moving from a single Infoblox HA pair to a failover set of geographically separated "Grid Member" HA pairs for even more DHCP fault-tolerance for our WAN users in case of a large network outage.

When moving to this setup, you must reconfigure every network and every DHCP range to "point" to the "failover set" consisting of a pairing between the two failover Infoblox Grid Members.
Since we have several hundred, DHCP Networks, my script was designed to dump to STDOUT all Networks and Ranges on the appliance before the change, then change all Networks/Ranges (save a few special setups I exempted) to point to the failover set, and then dump the "post operation" configuration to STDOUT for verification.

Please keep in mind that this script could definitely use a lot more catch statements for error handling, but I kept this pretty lean just to do this one job for me.

Without further adieu, see attached. Rename it from the .txt extension to .pl.

IBConvert Script

Tuesday, November 24, 2009

Adding a Cisco 3750 Switch to an Existing Stack

Another simple task that Cisco doesn't quite make 100% clear.

Let's say you have an existing Cisco 3750 Switch Stack and you have had a bit of an office redesign and you now need to add another member to the stack to add additional port capacity for the new users moving in.

This is mostly simple to do, but there are several checks to make along your journey, and hopefully I can point them out clearly.

  1. First, check the stacking status of your existing stack using the below command. It should say that the stack ring speed is "full". If it doesn't you need to ensure you have a complete stack "ring" thereby having redundant stacking paths on each switch.

    ZZStack#show switch stack-ring speed

    Stack Ring Speed : 32G
    Stack Ring Configuration: Full
    Stack Ring Protocol : StackWise

    See this publication for stack-wiring help:

  2. Once you know your stack is a complete ring("full") you can safely break this ring to insert your new switch. ENSURE YOUR NEW SWITCH IS POWERED OFF. Rack this new switch adjacent to your existing switches. "Break" the ring in one place so you can wire your new switch so it fits nice and neat in this new stack, following the wiring schema linked above. Once everything is wiring in place, you may NOW power on your new switch.

  3. After a few minutes your switch will have booted and the existing stack should recognize it as joining the stack. You can verify its status with:

    ZZStack#show switch stack-ring speed

    Stack Ring Speed : 32G
    Stack Ring Configuration: Full
    Stack Ring Protocol : StackWise

    ZZStack#show switch
    Switch/Stack Mac Address : 0024.9803.8e80
    H/W Current
    Switch# Role Mac Address Priority Version State
    *1 Master 0024.9803.8e80 15 0 Ready
    2 Member 0023.ac0f.7880 1 0 Ready
    3 Member 001a.e267.0080 1 0 Version Mismatch

    In this example switch 3 was added to the stack, the stack ring looks good, but on the switch status output, instead of saying "Ready" it says "Version Mismatch." You now need to ensure your switch gets the same IOS version as its stack-mates and reboots for this to take effect(see below).

    If your stack says your new switch is in the "Ready" state, you are in luck and you are done! (except for configuring your new user ports)

  4. Automatic Upgrade is a great thing in theory, but I've not had the best of luck with it. From what I can tell if your existing stack and this new member you are adding are running different IOS featuresets, the automatic upgrade will NOT WORK.

    To check to see if its working or not, check the log. If you see log entries resembling any of the below, it appears automatic upgrade is working as it should.
    ZZStack#show log
    Nov 24 16:58:36.388 EST: %IMAGEMGR-6-AUTO_COPY_SW_INITIATED: Auto-copy-software process initiated for switch number(s) 3

    This shows that it has started the auto upgrade process. You can check the status with the "show archive status" command or by continually checking the log.

    If you see the below it was successful and is now rebooting this member switch so the correct IOS loads and it can finally join the stack as it should(and you should be done):
    Nov 24 17:06:04.453 EST: %IMAGEMGR-6-AUTO_COPY_SW: Software successfully copied to
    Nov 24 17:06:04.453 EST: %IMAGEMGR-6-AUTO_COPY_SW: system(s) 3
    Nov 24 17:06:04.453 EST: %IMAGEMGR-6-AUTO_COPY_SW: Done copying software
    Nov 24 17:06:04.453 EST: %IMAGEMGR-6-AUTO_COPY_SW: Reloading system(s) 3

    If you don't see any of the above log entries, go on to the next step.

  5. This is the step they don't really explain well. Thankfully, the "Version Mismatch" state, while not activating any of its ports, does allow you to manipulate the flash filesystem of the inconsistent member so you can stage it from the main stack interface.

    If you made it to this step, it probably means there is a featureset mismatch(or some other problem) and you need to force this new member to take the IOS version that the stack is currently running. In this scenario, even the "archive copy-sw" command does not work, so you must either load the IOS bin file manually or use use the appropriate "archive download-sw" command with the "/allow-feature-upgrade" switch to load the IOS to the ENTIRE STACK again, including this new member(but you only need to reboot the new member). I prefer to use the archive command, its slow but its so easy!

    Here's the link to the software upgrade caveats/howtos for stack configurations(if it seems like I breezed through this last step):