I have one customer with standalone ESXi. Local Adaptec RAID, several LUNs on top of it, each with VMFS.
For some reasons ESXi hanged up. They rebooted ESXi and then strange thing happened.
One VMFS datastore just didn't come up.
What I've found in logs:
Aug 13 13:05:54 vmkernel: 0:00:00:39.189 cpu5:4101)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41000503c3c0) to NMP device "mpx.vmhba2:C0:T3:L0" failed on physical path "vmhba2:C0:T3:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0. Aug 13 13:05:54 vmkernel: 0:00:00:39.189 cpu5:4101)ScsiDeviceIO: 747: Command 0x28 to device "mpx.vmhba2:C0:T3:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0. Aug 13 13:05:54 vmkernel: 0:00:00:39.189 cpu5:4101)ScsiDeviceToken: 293: Sync IO 0x28 to device "mpx.vmhba2:C0:T3:L0" failed: I/O error H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0. Aug 13 13:05:54 vmkernel: 0:00:00:39.189 cpu7:4973)WARNING: FS3: 4053: Read error: I/O error Aug 13 13:05:54 vmkernel: 0:00:00:39.190 cpu7:4973)WARNING: FS3: 4820: Failed to initialize VMFS3 distributed locking on volume 4bb3ae52-50496cd0-bb39-003048c6e051: I/O error Aug 13 13:05:54 vmkernel: 0:00:00:39.190 cpu7:4973)FSS: 666: Failed to get object f530 28 1 4bb3ae52 50496cd0 3000bb39 51e0c648 0 0 0 0 0 0 0 :I/O error Aug 13 13:05:54 vmkernel: 0:00:00:39.190 cpu7:4973)OC: 941: Could not get object from FS driver: I/O error Aug 13 13:05:54 vmkernel: 0:00:00:39.190 cpu7:4973)WARNING: Fil3: 1930: Failed to reserve volume f530 28 1 4bb3ae52 50496cd0 3000bb39 51e0c648 0 0 0 0 0 0 0 Aug 13 13:05:54 vmkernel: 0:00:00:39.190 cpu7:4973)FSS: 666: Failed to get object f530 28 2 4bb3ae52 50496cd0 3000bb39 51e0c648 4 1 0 0 0 0 0 :I/O error Aug 13 13:05:54 vmkernel: 0:00:00:39.190 cpu7:4973)WARNING: FSS: 2946: Unable to reserve symlink f530 28 2 4bb3ae52 50496cd0 3000bb39 51e0c648 4 1 0 0 0 0 0 in OC Aug 13 13:05:54 vmkernel: 0:00:00:39.247 cpu5:4101)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x4100050794c0) to NMP device "mpx.vmhba2:C0:T3:L0" failed on physical path "vmhba2:C0:T3:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0. Aug 13 13:05:54 vmkernel: 0:00:00:39.247 cpu5:4101)ScsiDeviceIO: 747: Command 0x28 to device "mpx.vmhba2:C0:T3:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0. Aug 13 13:05:54 vmkernel: 0:00:00:39.247 cpu5:4101)ScsiDeviceToken: 293: Sync IO 0x28 to device "mpx.vmhba2:C0:T3:L0" failed: I/O error H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0x0.
I've checked VMFS signature placement, all is ok. And you can see that from UUID resolving for this volume.
I saw another customer with absolutely the same problem, but it happened on EMC SAN, so we just made snapshot and recovered VMs from there. Here is local RAID and no way to use the same solution.
Any thoughts?
---
MCSA, MCTS Hyper-V, VCP 3/4, VMware vExpert