Friday, April 18, 2014

Windows 2003 IPSec error blocks all traffic

This happened to one of our windows 2003 VMs this week - it was patched and rebooted and came up with no network traffic being passed.

The team first looked to revert to the previous known good state - the central storage snapshots.
But as it turns out these restored snapshots also came up with no network being passed.

I removed the old VM network adapter (AMD) and replaced it with the newer Intel, updated the vmwaretools, several reboots in between, all to no effect (the adapter was up and "connected") but not passing any traffic - could not ping the default gateway.

Finally found a post leading us to check the System Event logs

And finding a VMware KB article confirming the resolution:

Disable the Windows 2003 IPSec service and reboot - voila, traffic now unblocked!

Hope this saves other folks some time to resolution!

Tuesday, April 8, 2014

ESXi 5.5 vulnerable to OpenSSL Heartbleed bug - 4/19 Patch Released

Today the existence of a serious bug in OpenSSL was revealed which allows an attacker to read the memory of your system including usernames, passwords, keys etc.

More information here:

This site will check a URL for the vulnerability: 

This command line tool also checks for the vulnerability- its a python script you can use inside your firewalled zones.

Our ESXi 5.5 build 1331820 servers have the vulnerability.
And fully patched ESXi 5.5  build 1623387 (as of 4/8) shows the vulnerability as well.
I've posted in the vmware forums asking about the ETA of a patch to fix this in our VI.

Update 4/9: VMWare has supplied a KB of products affected 

Update 4/19 - Patch Released:

Note: VMWare recommends updating vCenter before ESXi

VMware KB: Resolving OpenSSL Heartbleed for ESXi 5.5 - CVE-2014-0160

Confirmed the vulnerability is removed by this patch - folks should also cycle keys and update passwords

cd /etc/vmware/ssl
chmod +t rui.crt
chmod +t rui.key
passwd root

Thursday, January 2, 2014

Team Canada Hockey Roster Map

One of my other passions (besides  virtualization, storage IOPS, cloud engineering  etc) is hockey.
Today Steve Yzerman, team Canada GM released the Team Canada Hockey roster for Sochi.
I wanted to create a visual, infographic map of the player's hometowns and found this cool geo mashup tool which is mostly intuitive and accurate.

View 2014 Team Canada Hometown map v0.3 in a full screen map

Monday, December 9, 2013

vNFS datastore for Cluster upgrades

In our pre 5.1 environment we have multiple clusters without centralized storage.
This is a problem when the cluster nodes need to be upgraded - we can not evacuate the nodes via vMotion since the VMs are on local disk.
Here is a solution: vNFS datastore provided you have adequate local disk on your hosts.

Present an NFS datastore to the cluster via a new VM using local storage.

1 - Configure a Centos/Linux VM with the local storage and share it via NFS
2 - Add the NFS datastore to the cluster nodes
3 - Storage vMotion off the VMs from the node to be upgraded one at a time (monitor the load on the NFS vm via top - the throughput is largely based on the disk IO and secondarily the network)
4 - vMotion off the VMs now they are on the NFS datastore
5 - put the host is maintenance mode and upgrade
6 - reverse the procedure to migrate the VMs back to the upgraded host and local storage

With 5.1 local to local storage vMotion and the upcoming vSAN offering this is less of an issue, but in the interim, its saved us much downtime.

Tuesday, October 22, 2013

vCenter 5.5 upgrade fails and rolls back

Upgrading our 5.1 vCenter to 5.5, we ran into this error:

"Simple Install Setup Wizard ended prematurely because of an error"

This turned out to be related to a mismatch btw the Registry VC IP address and the expected FQDN.
VMware KB2060511 provides the solution for this issue (clean up the failed upgrade dir, and modify the registry setting)