So, we had some drive failures on a box living at a colo facility, and over the course of the reboot and rebuild process, our install has fallen into "The ESXi host does not have persistent storage" error mode, and I haven't been able to recover the datastore.
The partition table still exists, and shows a vmfs partition:
~ # partedUtil getptbl /dev/disks/naa.6848f690ed12bc001a0aa0750395bd28
gpt
145782 255 63 2341994496
1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128
5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
2 1843200 10229759 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
3 10229760 2341994462 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
However, it will not mount using vmkfstools -V. I am seeing the following errors in vmkernel.log that are most likely related:
2019-05-23T22:50:31.187Z cpu4:6420)WARNING: LVM: 9998: Invalid firstPE 0
2019-05-23T22:50:31.187Z cpu4:6420)WARNING: LVM: 10005: Invalid lastPE 0
2019-05-23T22:50:31.187Z cpu4:6420)WARNING: LVM: 10017: Invalid volume state 0
2019-05-23T22:50:31.187Z cpu4:6420)WARNING: LVM: 5230: Error detected for vol , dev naa.6848f690ed12bc001a0aa0750395bd28:3
2019-05-23T22:50:31.187Z cpu4:6420)LVM: 7121: Device scan failed for <naa.6848f690ed12bc001a0aa0750395bd28:3>: Invalid metadata
Based on an XML config file I found, I know what the UUID of the old datastore was, but when I try to mount using it I get "No matching volume 5277e6a0-2d596de7-495f-b8ca3a5bd3a0 found!"
I'm asking the onsite techs if they can add an external drive, install Linux on that, and try to use vmfs-fuse, but is there anything else I can try using the ESXi CLI? The vast majority of the solutions I've found for lost datastores are for cases where the data is intact but the partition table is lost, but since the partition table still seems to be doing just fine, I'm not sure where to go from here.