We have a Dell T420 server running ESXi 5.5. The host has a single E5-2407 CPU (4 Core 2.2 GHz no Hyperthreading), 16 GB of RAM, and local RAID storage.
The host runs a single Windows Server 2008 VM, which was a P2V conversion. The VM is currently assigned 1 Socket and 4 Cores, matching the host CPU.
Since setting up this server about a year ago, it has had a recurring problem pop up every few months. When the problem happens, the Server 2008 VM suddenly has max CPU usage and terrible performance, but CPU is being maxed not by a single program, but from several normal processes. For example, WMI, explorer.exe, and task manager may all show 30% CPU usage. Meanwhile, though the Guest VM has 100% CPU usage, the Host will show it only using 300-500 MHz of CPU, or about 7% of available. This is despite the fact it's the sole VM and has full reign of all the Host resources. So the problem appears to be not with abnormal load on the Guest VM, but under-availability of Host resources.
Restarting the VM does not correct the issue, and the restart, which normally only takes a few minutes, takes 30 minutes. In the past, restarting the ESXi host has temporarily fixed the issue, however it popup up again today and is persisting even after several ESXi host restarts. See attached screenshots for the settings and symptoms:
This is the performance monitor of the VM while the problem is happening. Note that on the VM, Windows 2008 is reporting 100% CPU usage of a 4 core 2.2 GHz CPU, while on the Host side, the CPU usage never passes 626 MHz.
Here are the state counters while the problem is happening. During boot of the VM, the WAIT and IDLE counters were pretty high, but they finally dip back down when boot up is finished.
Here are the settings and stats of the ESXi host.
Here are the settings and stats of the guest VM while the problem is happening.
Other troubleshooting steps:
We changed vCPU count on the VM from 4 to 1, no change.
We set CPU reserve for the VM to 50%, and to 100%, no change.
The VM had been version 8 and we updated to version 9, no change.
Has anyone ever run into this problem? Any help toward a solution would be appreciated.