Hey there,
we got the same problem two weeks ago. One of our Datastores was suddenly unavailable.
Back then there where machines running on that Datastore.
We rebooted the server, the Datastore was back on but we had serious problems with locked files (which still isn't resolved).
See the Thread here: http://communities.vmware.com/thread/319421?tstart=150
We now are in fear that there might be more locked files or some more trouble when rebooting the server.
We also would like to know the cause of that Datastore beeing unavailable.
We already tried "vmkfstools -V" and looking at the problem with esxcfg-scsidevs -l / -a / -m (see output below!)
I was deleting some old backups of machines when the Datastore vanished (deleting some machines via the Console/SSH on the same Datastore worked a few minutes before).
I include the Console output at the time the Datastore was gone (see below).
Maybe someone has some useful tips on how to remount the Datastore / get it back online or can indentify the source of the problem!
Thanks in advance for replies, asgaroth
/vmfs/volumes/4bd88e21-e8b0d6ea-6c1a-003048c30e8c # rm -rf serverxyz.subdomain.domain rm: cannot remove 'serverxyz.subdomain.domain/serverxyz.subdomain.domain-flat.vmdk': Resource temporarily unavailable rm: cannot stat 'serverxyz.subdomain.domain/serverxyz.subdomain.domain.vmdk': Device or resource busy rm: cannot stat 'serverxyz.subdomain.domain/serverxyz.subdomain.domain.nvram': Device or resource busy rm: cannot stat 'serverxyz.subdomain.domain/vmware-37.log': Device or resource busy rm: cannot stat 'serverxyz.subdomain.domain/vmware-38.log': Device or resource busy rm: cannot stat 'serverxyz.subdomain.domain/vmware-39.log': Device or resource busy rm: cannot stat 'serverxyz.subdomain.domain/serverxyz.subdomain.domain-c1309193.hlog': Device or resource busy rm: cannot stat 'serverxyz.subdomain.domain/vmware-42.log': Device or resource busy rm: cannot stat 'serverxyz.subdomain.domain/vmware-41.log': Device or resource busy rm: cannot stat 'serverxyz.subdomain.domain/vmware-40.log': Device or resource busy rm: cannot stat 'serverxyz.subdomain.domain/vmware.log': Device or resource busy rm: cannot remove 'serverxyz.subdomain.domain': Device or resource busy ash: getcwd: Device or resource busy (unknown) # ls -al ls: .: Input/output error ash: getcwd: Input/output error (unknown) # cd / ~ # ls -al /vmfs/volumes/ ls: /vmfs/volumes/4bd88e21-e8b0d6ea-6c1a-003048c30e8c: Input/output error drwxr-xr-x 1 root root 512 Jul 12 13:37 . drwxr-xr-x 1 root root 512 Jun 28 15:57 .. drwxr-xr-x 1 root root 8 Jan 1 1970 386a41e4-ff1587ba-1385-2feaeed494ad drwxr-xr-t 1 root root 1960 Jul 12 12:56 4bc8414f-3206fe54-f06e-003048c30e8c drwxr-xr-t 1 root root 3500 Jul 8 22:30 4bd88e99-c67ad631-6c62-003048c30e8c drwxr-xr-t 1 root root 980 May 21 14:43 4dd7cf77-c3e57835-e9ef-003048c30e8c drwxr-xr-x 1 root root 8 Jan 1 1970 8c7e2fe8-f4c58d9f-34d3-8553ffee79a0 lrwxr-xr-x 1 root root 35 Jul 12 13:37 Datastore1 -> 4bc8414f-3206fe54-f06e-003048c30e8c lrwxr-xr-x 1 root root 35 Jul 12 13:37 Datastore2 -> 4bd88e21-e8b0d6ea-6c1a-003048c30e8c lrwxr-xr-x 1 root root 35 Jul 12 13:37 Datastore3 -> 4bd88e99-c67ad631-6c62-003048c30e8c lrwxr-xr-x 1 root root 35 Jul 12 13:37 Datastore4 -> 4dd7cf77-c3e57835-e9ef-003048c30e8c lrwxr-xr-x 1 root root 35 Jul 12 13:37 Hypervisor1 -> 386a41e4-ff1587ba-1385-2feaeed494ad lrwxr-xr-x 1 root root 35 Jul 12 13:37 Hypervisor2 -> 8c7e2fe8-f4c58d9f-34d3-8553ffee79a0 lrwxr-xr-x 1 root root 35 Jul 12 13:37 Hypervisor3 -> e00f98e1-2bcc0c91-e7a2-3487611c1557 drwxr-xr-x 1 root root 8 Jan 1 1970 e00f98e1-2bcc0c91-e7a2-3487611c1557 ~ # ls -al /vmfs/volumes/Datastore2 lrwxr-xr-x 1 root root 35 Jul 12 13:38 /vmfs/volumes/Datastore2 -> 4bd88e21-e8b0d6ea-6c1a-003048c30e8c ~ # ls -al /vmfs/volumes/Datastore2/ ls: /vmfs/volumes/Datastore2/: Input/output error ~ # df Filesystem 1k-blocks Used Available Use% Mounted on visorfs 1338636 267996 1070640 20% / vmfs3 488112128 328402944 159709184 67% /vmfs/volumes/4bc8414f-3206fe54-f06e-003048c30e8c nfs 315829740 135465900 180363840 43% /vmfs/volumes/0d6becf5-2ea0d633 nfs 1175588244 230332944 945255300 20% /vmfs/volumes/fa828cd2-adbc3d72 vmfs3 976486400 578560 975907840 0% /vmfs/volumes/4dd7cf77-c3e57835-e9ef-003048c30e8c vfat 292752 139328 153424 48% /vmfs/volumes/e00f98e1-2bcc0c91-e7a2-3487611c1557 vfat 255716 90204 165512 35% /vmfs/volumes/8c7e2fe8-f4c58d9f-34d3-8553ffee79a0 vfat 255716 61804 193912 24% /vmfs/volumes/386a41e4-ff1587ba-1385-2feaeed494ad ~ # vmkfstools -V ~ # ls /vmfs/volumes/ 0d6becf5-2ea0d633 4dd7cf77-c3e57835-e9ef-003048c30e8c Datastore4 e00f98e1-2bcc0c91-e7a2-3487611c1557 386a41e4-ff1587ba-1385-2feaeed494ad 8c7e2fe8-f4c58d9f-34d3-8553ffee79a0 Hypervisor1 fa828cd2-adbc3d72 4bc8414f-3206fe54-f06e-003048c30e8c Datastore1 Hypervisor2 nfs-vmds1 4bd88e99-c67ad631-6c62-003048c30e8c Datastore3 Hypervisor3 nfs-vmds2 ~ # ls -l /vmfs/volumes/ drwxr-x--- 1 root root 4096 Jun 30 12:47 0d6becf5-2ea0d633 drwxr-xr-x 1 root root 8 Jan 1 1970 386a41e4-ff1587ba-1385-2feaeed494ad drwxr-xr-t 1 root root 1960 Jul 12 12:56 4bc8414f-3206fe54-f06e-003048c30e8c drwxr-xr-t 1 root root 3500 Jul 8 22:30 4bd88e99-c67ad631-6c62-003048c30e8c drwxr-xr-t 1 root root 980 May 21 14:43 4dd7cf77-c3e57835-e9ef-003048c30e8c drwxr-xr-x 1 root root 8 Jan 1 1970 8c7e2fe8-f4c58d9f-34d3-8553ffee79a0 lrwxr-xr-x 1 root root 35 Jul 12 13:59 Datastore1 -> 4bc8414f-3206fe54-f06e-003048c30e8c lrwxr-xr-x 1 root root 35 Jul 12 13:59 Datastore3 -> 4bd88e99-c67ad631-6c62-003048c30e8c lrwxr-xr-x 1 root root 35 Jul 12 13:59 Datastore4 -> 4dd7cf77-c3e57835-e9ef-003048c30e8c lrwxr-xr-x 1 root root 35 Jul 12 13:59 Hypervisor1 -> 386a41e4-ff1587ba-1385-2feaeed494ad lrwxr-xr-x 1 root root 35 Jul 12 13:59 Hypervisor2 -> 8c7e2fe8-f4c58d9f-34d3-8553ffee79a0 lrwxr-xr-x 1 root root 35 Jul 12 13:59 Hypervisor3 -> e00f98e1-2bcc0c91-e7a2-3487611c1557 drwxr-xr-x 1 root root 8 Jan 1 1970 e00f98e1-2bcc0c91-e7a2-3487611c1557 drwxr-x--- 1 root root 4096 Jun 30 12:36 fa828cd2-adbc3d72 lrwxr-xr-x 1 root root 17 Jul 12 13:59 nfs-vmds1 -> 0d6becf5-2ea0d633 lrwxr-xr-x 1 root root 17 Jul 12 13:59 nfs-vmds2 -> fa828cd2-adbc3d72 ~ # esxcfg-scsidevs -l mpx.vmhba0:C0:T0:L0 Device Type: CD-ROM Size: 0 MB Display Name: Local TSSTcorp CD-ROM (mpx.vmhba0:C0:T0:L0) Multipath Plugin: NMP Console Device: /vmfs/devices/cdrom/mpx.vmhba0:C0:T0:L0 Devfs Path: /vmfs/devices/cdrom/mpx.vmhba0:C0:T0:L0 Vendor: TSSTcorp Model: CDDVDW SH-S202N Revis: SB01 SCSI Level: 5 Is Pseudo: false Status: on Is RDM Capable: false Is Removable: true Is Local: true Other Names: vml.0005000000766d686261303a303a30 VAAI Status: unknown mpx.vmhba32:C0:T0:L0 Device Type: Direct-Access Size: 998 MB Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0) Multipath Plugin: NMP Console Device: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0 Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0 Vendor: TinyDisk Model: 2005-03-10 Revis: 2.00 SCSI Level: 2 Is Pseudo: false Status: on Is RDM Capable: false Is Removable: true Is Local: true Other Names: vml.0000000000766d68626133323a303a30 VAAI Status: unknown naa.600050e0d7beda002095000037530000 Device Type: Direct-Access Size: 953664 MB Display Name: Local AMCC Disk (naa.600050e0d7beda002095000037530000) Multipath Plugin: NMP Console Device: /vmfs/devices/disks/naa.600050e0d7beda002095000037530000 Devfs Path: /vmfs/devices/disks/naa.600050e0d7beda002095000037530000 Vendor: AMCC Model: 9650SE-8LP DISK Revis: 4.10 SCSI Level: 5 Is Pseudo: false Status: on Is RDM Capable: false Is Removable: false Is Local: true Other Names: vml.0200000000600050e0d7beda002095000037530000393635305345 VAAI Status: unknown t10.AMCC____9QJ0ZCFYD87EFE007FC4 Device Type: Direct-Access Size: 953664 MB Display Name: Local AMCC Disk (t10.AMCC____9QJ0ZCFYD87EFE007FC4) Multipath Plugin: NMP Console Device: /vmfs/devices/disks/t10.AMCC____9QJ0ZCFYD87EFE007FC4 Devfs Path: /vmfs/devices/disks/t10.AMCC____9QJ0ZCFYD87EFE007FC4 Vendor: AMCC Model: 9650SE-8LP DISK Revis: 4.10 SCSI Level: 5 Is Pseudo: false Status: on Is RDM Capable: false Is Removable: false Is Local: true Other Names: vml.010000000039514a305a434659443837454645303037464334393635305345 VAAI Status: unknown t10.AMCC____9QJ16PFCD87EFE00CB5E Device Type: Direct-Access Size: 953664 MB Display Name: Local AMCC Disk (t10.AMCC____9QJ16PFCD87EFE00CB5E) Multipath Plugin: NMP Console Device: /vmfs/devices/disks/t10.AMCC____9QJ16PFCD87EFE00CB5E Devfs Path: /vmfs/devices/disks/t10.AMCC____9QJ16PFCD87EFE00CB5E Vendor: AMCC Model: 9650SE-8LP DISK Revis: 4.10 SCSI Level: 5 Is Pseudo: false Status: on Is RDM Capable: false Is Removable: false Is Local: true Other Names: vml.010000000039514a3136504643443837454645303043423545393635305345 VAAI Status: unknown t10.AMCC____U34961226AEF5C004A68 Device Type: Direct-Access Size: 476827 MB Display Name: Local AMCC Disk (t10.AMCC____U34961226AEF5C004A68) Multipath Plugin: NMP Console Device: /vmfs/devices/disks/t10.AMCC____U34961226AEF5C004A68 Devfs Path: /vmfs/devices/disks/t10.AMCC____U34961226AEF5C004A68 Vendor: AMCC Model: 9650SE-8LP DISK Revis: 4.10 SCSI Level: 5 Is Pseudo: false Status: on Is RDM Capable: false Is Removable: false Is Local: true Other Names: vml.01000000005533343936313232364145463543303034413638393635305345 VAAI Status: unknown ~ # esxcfg-scsidevs -a vmhba0 ata_piix link-n/a ide.vmhba0 (0:0:31.1) Intel Corporation 631xESB/632xESB IDE Controller vmhba1 3w-9xxx link-n/a unknown.vmhba1 (0:1:0.0) 3ware Inc 3ware 9650SE vmhba32 usb-storage link-n/a usb.vmhba32 () USB vmhba33 ata_piix link-n/a ide.vmhba33 (0:0:31.1) Intel Corporation 631xESB/632xESB IDE Controller ~ # esxcfg-scsidevs -m t10.AMCC____U34961226AEF5C004A68:1 /vmfs/devices/disks/t10.AMCC____U34961226AEF5C004A68:1 4bc8414f-3206fe54-f06e-003048c30e8c 0 Datastore1 naa.600050e0d7beda002095000037530000:1 /vmfs/devices/disks/naa.600050e0d7beda002095000037530000:1 4dd7cf77-c3e57835-e9ef-003048c30e8c 0 Datastore4 t10.AMCC____9QJ16PFCD87EFE00CB5E:1 /vmfs/devices/disks/t10.AMCC____9QJ16PFCD87EFE00CB5E:1 4bd88e99-c67ad631-