Someone recently brought a Dell PE-R720 to me after a "crash" and having someone else replace a failed hard drive. From what I have been told, it was a simple drive failure and replacement. However, after the replacement, when the server booted, none of the VM's came up and they are all listed as inaccessible in the vSphere client. The system has five 4TB SATA drives in it, configured RAID 5, and two volumes on that RAID5 - 500GB and 14ish TB. No, apparently the office this came from doesn't have any backups, which is why it has ended up with me.
Inside vSphere, the 500GB device has a datastore that does show up. The 14TB device has a datastore as well, but it does not show up. I can look at Devices, and see both of them, and the associated VMFS partitions on them.
When I connect via SSH, I can see and navigate both VMFS volumes:
/vmfs/volumes # ls -la
total 3076
drwxr-xr-x 1 root root 512 Sep 14 04:11 .
drwxr-xr-x 1 root root 512 Sep 13 14:53 ..
drwxr-xr-x 1 root root 8 Jan 1 1970 560948da-a221fd2d-163f-f8bc1246cd0a
drwxr-xr-t 1 root root 1260 Sep 28 2015 560948e3-e9884764-70d0-f8bc1246cd0a
drwxr-xr-x 1 root root 8 Jan 1 1970 560948e6-4e07cd8b-51dc-f8bc1246cd0a
drwxr-xr-t 1 root root 2380 Nov 23 2015 560954f5-debd2d16-560c-f8bc1246cd0a
drwxr-xr-x 1 root root 8 Jan 1 1970 82245d7c-3eb4aa98-e653-e1cfc32d3ff8
lrwxr-xr-x 1 root root 35 Sep 14 04:11 datastore1 -> 560948e3-e9884764-70d0-f8bc1246cd0a
drwxr-xr-x 1 root root 8 Jan 1 1970 e3cd597c-6b667b0d-70c6-4fd873e04ddd
lrwxr-xr-x 1 root root 35 Sep 14 04:11 storage -> 560954f5-debd2d16-560c-f8bc1246cd0a
If I do a Rescan All from vSphere, I can see the following in the hostd.log file:
2017-09-14T04:13:16.989Z [7E6C2B70 error 'Hostsvc.FSVolumeProvider' opID=CA63A857-0000027E user=root] RefreshVMFSVolumes: ProcessVmfs threw HostCtlException Unable to get FS Attrs for /vmfs/volumes/560954f5-debd2d16-560c-f8bc1246cd0a
VmFileSystem: SlowRefresh() failed: Unable to get FS Attrs for /vmfs/volumes/560954f5-debd2d16-560c-f8bc1246cd0a. Unable to get FS Attrs for /vmfs/volumes/560954f5-debd2d16-560c-f8bc1246cd0a
2017-09-14T04:13:17.333Z [7E681B70 error 'Hostsvc.FSVolumeProvider' opID=CA63A857-00000280 user=root] RefreshVMFSVolumes: ProcessVmfs threw HostCtlException Unable to get FS Attrs for /vmfs/volumes/560954f5-debd2d16-560c-f8bc1246cd0a
I appear to be able to copy the data files off of this volume using WinSCP (or at least it is in the middle of a multi-hour copy with no issues so far), so I'm guessing the volume header was messed up somehow in the rebuild, or I wasn't told everything that happened.
Is there any way I can get this VMFS volume back up and functional without hours and hours of copying off data and then recreating the volume and copying the data back? Or is this a case where a single support incident with VMware may be the only resort?