We have been trialing Dell Rapid-Recovery for ESXi backups, and occasionally experience consolidation failures. Any advice on how to track down why, so that we can fix it?
This server is running ESXi 5.5 build 2068190.
Here is some info from hostd.log:
2016-07-21T09:00:16.847Z [52E80B70 info 'Vimsvc.TaskManager' opID=hostd-854b user=root] Task Created : haTask-4-vim.VirtualMachine.consolidateDisks-344057184
2016-07-21T09:00:16.848Z [51080B70 info 'Vmsvc.vm:/vmfs/volumes/548c3c98-2367a4b4-9aa2-0025908c25f8/domain1/domain1.vmx' opID=hostd-854b user=root] State Transition (VM_STATE_ON -> VM_STATE_CONSOLIDATE_ALL_DISKS)
...
(Lots of verbose messages that do not appear to have anything to do with consolidation)
...
2016-07-21T09:00:18.740Z [4F5C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/548c3c98-2367a4b4-9aa2-0025908c25f8/domain1/domain1.vmx'] Consolidate Disks translated error to vim.fault.FileLocked
2016-07-21T09:00:18.740Z [4F5C1B70 info 'Vmsvc.vm:/vmfs/volumes/548c3c98-2367a4b4-9aa2-0025908c25f8/domain1/domain1.vmx'] Consolidate Disks failed: vim.fault.FileLocked
2016-07-21T09:00:18.740Z [4F5C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/548c3c98-2367a4b4-9aa2-0025908c25f8/domain1/domain1.vmx'] Consolidate Disks message: An error occurred while consolidating disks: Failed to lock the file.
-->
2016-07-21T09:00:18.740Z [4F181B70 info 'Vimsvc.ha-eventmgr'] Event 7495 : Virtual machine domain1 disks consolidation failed on vsphere1 in cluster vsphere1 in ha-datacenter.
2016-07-21T09:00:18.742Z [4F181B70 verbose 'Vmsvc.vm:/vmfs/volumes/548c3c98-2367a4b4-9aa2-0025908c25f8/domain1/domain1.vmx'] Time to gather Snapshot information ( read from disk, build tree): 1 msecs. needConsolidate is true.
2016-07-21T09:00:18.742Z [4F181B70 verbose 'Vmsvc.vm:/vmfs/volumes/548c3c98-2367a4b4-9aa2-0025908c25f8/domain1/domain1.vmx'] Snapshot property update: Configure will be invalidated for:
2016-07-21T09:00:18.758Z [4F181B70 verbose 'Vmsvc.vm:/vmfs/volumes/548c3c98-2367a4b4-9aa2-0025908c25f8/domain1/domain1.vmx'] Time to gather config: 15 (msecs)
This is the third time it has done this, each time with a different guest / vmdk. I'm at a loss on how to proceed. The only fix that has worked on the prior occasions was to reboot the ESXi host and do a manual consolidation.
The vSphere Client and command-line tools all give similar errors when attempting to consolidate without a reboot, stating that the they are unable to access the file since it is locked. Even restarting the hostd daemon is not sufficient to allow consolidation to proceed -- nothing but a full host reboot.
Any ideas how to proceed?
Thanks in advance