We currently have ESX host version 6.7 update 1. There is a flooding ATS miscompare messages whenever the storage array is under heavy load. siDeviceIO: 3082: Cmd(0x459a60604580) 0x89, CmdSN 0x3cc83 from world 2171298 to dev x failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.
I have read these articles https://cormachogan.com/2017/08/24/ats-miscompare-revisited-vsphere-6-5/ and VMware Knowledge Base to understand more about why miscompares happen. What I understand is this (and I might be completely wrong):
1. ATS tests an in-memory value with an on-disk value on the storage array. From what I have read, I think the value being written is probably a timestamp. We see miscomapare messages if there's a mismatch between these values.
2. Each ESX host has a separate heartbeat region which has to be updated every 3 seconds to maintain the lock on a specific region of the volume or disk. So, two hosts can never update the same heartbeat region though they can observe the lock, break the lock, etc.
My question is this - How can a heavily loaded array lead to a miscompare? These are the following scenarios I can think of:
1. If the ATS operation times out, then the host would know that the operation has timed out. It will retry with the same ATS image, and it should ultimately succeed. If the operation has succeeded on the storage array but the host had timed out, the next ATS would be a false miscompare, and the operation would eventually succeed.
2. If the host had timed out, and the operations hadn't succeeded on the storage array, then the retry would the timed out test pattern itself would succeed.
I am not able to come up with a situation where we can see a MISCOMPARE. To be honest, after thinking out loud, I don't I can come up with any scenario when we can see a miscompare. Can anyone explain the exact purpose of a MISCOMPARE or lead me to any documents that can explain it further?