vMotion High CPU vmmemctl

I was investigating a strange issue today during a vMotion of a CentOS 5 tomcat server (running 6 tomcat containers)

source host: Dell R900 Intel Xeon E7450 ESX 4.1 260247

dest host: Dell R610 Xeon 5680 ESXi 4.1 260247 (new)

I vMotion'ed the first of 2 Centos 5 VMs from the source to dest while running top and observed an unusual lag in top refresh during the cutover.

after about 15 seconds of no refreshes the VM came back and top showed a high load average with the vmmemctl near the top. I also observed the "mem used" column on this 8Gb VM went from 7.9Gb to 2.4Gb in the span of 10-40 seconds!

It turned out (after examining esxtop->(m)emory output) that the VM was swapping! (eventhough we were no where near overcommitted on these ESX hosts.)

The SWCUR and SWTGT columns showed zeros for all other behaving VMs, but these two app servers had recently had their memory allocation increased from 2 Gb to 8Gb - but the
Edit Settings ->Resources->Memory-> Unlimited checkbox
had not retained its unlimited value.
Updating it to be unlimited and restarting the VM fixed the issue.