I have 1 VM that has a significant %lat_c/CPU Latency and it makes no sense.
I have found that CPU latency is made up of Ready, Swap Wait, and Power control issues. But all of these are under control:
No Swap wait at all, very low ready, and power management is fully off. (C-state and CE1 are both off in the bios and not reporting in esxtop) (Power is set for OS Control, esxi is set at High Performance and esxtop is reporting all CPUs at 100% P0 level) So it can not be power related. It is just this one VM. Not other VMs are showing this level of latency. Host totals are:
There are only 10 VMs on this host, 16 vCPUs, 704 worlds. The host is a dual Hex-core with hyperthreading (24 threads possible), 192GB of RAM. It is not the newest thing in the world, it is 2.5GHz Westmere processors. It is running ESXi 5.5 (without R1 so far). We have run it on a v55R1 host with the same results.
The VM is 80GB RAM and Dual proc averaging 65% to 68% CPU with a few spikes to 100%. So it really does need 2 procs and we are monitoring it to see when it will need a third. It is an SQL server and is using all 80GB of RAM and could actually use more. It is hammering the SAN pretty hard reading indexes (99% reads from SSD cache on the SAN). Active Memory is averaging over 70 GB. The entire hosts active memory is barely over 80GB. The other VMs are all pretty small.
What else could be causing 10% CPU Latency. Does this really mean that this VM is waiting for physical CPUs 10% of the time when it should be doing work? The performance of the MS CRM Dynamics application is not what we would like. We have been contemplating adding an SSD to the host to use vFlash Cache to bring the entire 350GB database to the host to offload the reads from the SAN, but while that would offload the SAN we don't really figure it will help performance of this VM that much since we are seeing nearly 100% of the reads coming from the SSD cache that is on the Nimble SAN.
Can anyone give me any idea what is causing this latency and how to resolve it.