Thursday, March 31, 2011

Upgrade vCenter SQL 2005 Express to SQL 2008 Standard

vCenter comes with SQL Server 2005 Express - works fine for smaller environments, but after a while your environment (# ESX servers and # of VMs) will grow and you will exceed the 4Gb licensed limit for this "free" SQL 2005 Express instance.

Once exceeded, your vCenter service will crash and fail on restart.

The error in your vCenter server vpxd log file will look like:

Error inserting events: "ODBC error: (42000) - [Microsoft][SQL Native Client][SQL Server]Could not allocate space for object 'dbo.VPX_EVENT_ARG'.'PK_VPX_EVENT_ARG' in database 'VIM_VCDB' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup." is returned when executing SQL statement "INSERT INTO VPX_EVENT_ARG


you can buy yourself some space by purging old records - but to address the root cause I chose to upgrade to a full SQL Server 2008.
Below are the steps to accomplish a vCenter SQL Server 2005 Express upgrade to SQL Server 2008 Standard (casting off the 4Gb limit). My vCenter 4.1 U1 runs as a VM in Windows 2008 64-bit Standard guest OS. Since the vCenter runs as a VM I forked off a clone to test the upgrade steps while not affecting the production vCenter. The upgrade path I used: SQL 2005 Express -> SQL 2008 Express -> SQL 2008 Standard:

SQL 2005 Express -> SQL 2008 Express:

1) Free up 5-10Gb on your vCenter server - windirstat is one of my favorite tools for identifying what is eating space on windows VMs
2) Take a snapshot of your vCenter VM "preSQL upgrade" (just in case)
3) download SQL 2008 Express: http://www.microsoft.com/express/Database/
4) run the SQLEXPRWT_x64_ENU.exe to upgrade to SQL 2008 Express
5) You will likely get an error towards the end:
error on SQL Server 2005 tools - “Please remove” -

To Re-run with success, you need to rename the registry entry:
HKLM\SOFTWARE\Wow6432Node\Microsoft\Microsoft SQL Server\90\Tools out of the way (eg HKLM\SOFTWARE\Wow6432Node\Microsoft\Microsoft SQL Server\90\Tools_old)
6) once the registry entry is renamed, click the re-run button to retry without restarting the upgrade

SQL 2008 Express -> SQL 2008 Standard:

1) Purchase a SQL Server 2008 Standard license to obtain a valid product code
2) Download the SQL Server 2008 Standard install
3) run the setup.exe with a special parameter "SKUUPGRADE=1" from the cmd prompt
4) Don't choose upgrade existing - choose "New install or add features to existing..." for the type of install.
5) Select all features when prompted
5) Supply the credentials for SQL services
6) This will run for a good 30+ minutes.
7) At the end your vCenter service will fail to start with the following in the log:

Failed to create https proxy: An attempt was made to access a socket in a way forbidden by its access permissions.


8) Turns out (http://kb.vmware.com/kb/1026305 clued me into this) SQL 2008 brings a reporting service on port 80 - CONFLICTING with vCenter! - disable the reporting service
9) restart vCenter (I rebooted to ensure clean startup)
10) connect with VI Client to verify all is good!
11) Once the vCenter functions are all verified remember to delete the preSQL snapshot

While figuring out how to jump through these upgrade hoops I was wishing for a MySQL option for vCenter - apparently in 2009 VMware attempted a beta of this, but it was abandonded - today the options are SQL or Oracle for the vCenter DB.

UPDATE 4/4/11:
I noticed the SQLEXP_VIM database was not migrated to the SQL Server 2008 instance.
I needed to detach it from the 2005 instance and re-attach it to the 2008 instance according to:
http://get-admin.com/blog/how-to/migrate-vcenter-database-from-sql-express-to-full-sql-server/

Once this was done, we were finally rid of the 4Gb limit on the vCenter SQL DB!

32 bit DSN required by Update Manager:
per
http://blog.eight02.com/2010/11/bits-of-odbc-advice-for-vcenter-update.html
"The DSN, ‘VUM’ does not exist or is not a 32 bit system DSN. Update Manager requires a 32 bit system DSN."

Cause: Using the ODBC tool in the Control Panel will create a 64-bit DSN. You need to use the 32-bit ODBC tool which is located at C:\Windows\SysWOW64\odbcad32.exe. Do NOT use the odbcad32.exe located in the C:\Windows\System32 folder. While it has the same file name, they are two different files!! - I was running the wrong exe by default...whew!

fix domain errors with newsid.exe

Maybe you are a unix guy like me and you sometimes forget when cloning one-off temporary Windows VM's for testing to run sysprep or newsid.exe to avoid errors like:

The system cannot log you on due to the following error: The name or
security ID (SID) of the domain specified is inconsistent with the
trust information for that domain.
The officially supported method is of course to use sysprep (perfect for the cloning from a prepared template use case)

But for the other use cases (eg testing a P2V or an upgrade of VC VM etc) where official support is not as important (the cloned VM will be deleted once testing is done), newsid works just fine.

Microsoft documents link to sysprep, but here is the small utility downloadable for posterity

newsid.exe

Again, this is unsupported by microsoft - use sysprep for production use

Wednesday, March 23, 2011

vmware-tools install error

While installing/updating vmware tools (VMwareTools-8.3.2-257589.tar.gz) for linux guests, I sometimes receive the error:

Unable to create symbolic link "/usr/lib/vmware-tools/bin"

or

Unable to create symbolic link "/usr/lib/vmware-tools/libconf"

Seems the installer has a bug where it fails to remove the existing directory before creating the symlink (it prompts to overwrite, but of course overwriting a dir with a symlink is not possible - therefore I'd call this a BUG)

The solution is to simply remove the existing dirs:

rm -rf /usr/lib/vmware-tools/libconf
rm -rf /usr/lib/vmware-tools/bin

and run the install again:

./vmware-install.pl -d

Monday, March 14, 2011

Apache Optimal vCPU Analysis

Last week I posed a question in the VMware forums:

How to determine optimal vCPU for apache workloads given a specified hardware and software configuration?

I prefaced the question by stating we strictly adhere to the best practice of keeping the vCPU at 1 unless the workload is multithreaded and capable of benefitting from the additional vCPUs.
Given the highly multithreaded nature of apache we had set the vCPU to 8, but without any numbers to quantify this was the optimal value this was more of an intuitive configuration based on our workloads and knowledge of ESX.

Without any feedback in the week following the posting, I took it as an opportunity to design an experiment to measure the effect of varying the vCPU on apache throughput, latency, response time etc.

What follows are some unexpected observations that may or may not be useful to others looking at tuning vCPU for their environments.

Experiment design:
The goal of this experiment is to measure the effects of varying the vCPU setting of a CentOS 5 Apache webserver VM. For generating the web server load I chose apachebench.


setup:
cloned a production web server for testing (changing only its IP)
Config:

apache version 2.2.3

running in Centos5 VM

with 8Gb RAM allocated

Threads (via scoreboard) range from 50-100 active (with max 300)

vSphere ESXi 4.1 U1 (build 348481)

Hardware: Dell R610 with 2 x 6 Core Intel Xeon X5680 CPUS @ 3.3GHz


Starting with a vCPU setting of 1, I ran the following script to iterate from 25 to 175 concurrent requests in increments of 25 - (the URL was an average page of 50K and the 10000 requests took about 1 minute to serve total):

#!/bin/csh
# Script to increase concurrent requests

set x = 25
while ( $x -lt 200)
ab -n 10000 -c $x http://web-06/

@ x+=25
end

The output for each vCPU# run was captured, then the VM was brought down to increase the vCPU setting to 2, 4, 6, 8 - capturing the results for each of the 5 vCPU levels. Bringing the results into excel tables the following metrics were compared across the vCPU runs (and I re-ran the vCPU runs later to confirm the data per vCPU config was not varying wildly from run to run) :

Total Connect Time:
This is the total time (Connect, Processing, Waiting) in milliseconds spent serving the request (we want this to be as low as possible. Observe
the 1 vCPU configuration is markedly higher than the 2, 4, 6, 8 configurations.




Deviation: The following apache bench metric accounts for how variable the total time to serve is - ideally we want this to be as small as possible so our apache performance is deterministically consistent. Observe how the 2 and 4 vCPU deviation is markedly higher than the 1 and 8 vCPU configurations (y-axis is milliseconds deviation from mean)



Requests per second: This metric measures the average requests per second served (x-axis is requests served per second). Below we can see the 2 and 4 vCPU configurations outperform the rest, but we have to remember this comes at the expense of increased variability. We are beginning to see the tradeoff: the 2 or 4 vCPU config will give most users slightly better response times, but the 8 vCPU config's behavior is more deterministic and gives a more consistently decent response time.




Throughput
: This metric measures the Kb/sec delivered by the apache instance at each vCPU setting. It mirrors the requests/sec metric - 2 and 4 vCPU configs deliver higher throughput on average, but at the expense of higher variability - giving some requests much lower throughput.






Conclusions:
At our average peak thread load (call it 75 requests/second per apache instance) we see that while the 4 vCPU config will deliver 9.4% higher throughput ON AVERAGE, than the 8 vCPU config, the 8 vCPU config provides 45.3% better (lower) variability from that average response. All other things equal, the decision to keep the 8 vCPU configuration for our apache VM instances is therefore easily rationalized in the context of giving most users a slightly lower throughput while guaranteeing their worst response will be much better with the 8 vCPU vs the 4vCPU config.