Hi,
Need help in troubleshooting an issue where I lose connectivity to iSCSI LUN.
Environment (used for lab)
Two hosts ESXI (Version 5,5 and 6.0). vCenter version 6.5.
Storage – QNAP (RAID 6) 8 disks with iSCSI LUN.
Symptom
Every one to two weeks I lose all my virtual machines on both hosts. Reboot of the actual machines and started all virtual machine has been the solution so fare.
On both hosts I can see these messages several times per day
Device naa.600140537fc0a26dc759d4634d91b5d7 performance has improved. I/O latency reduced from 20369 microseconds to 6604 microseconds. Info 2018-12-31 06:19:25
Lost access to volume 503f2cab-3901da70-efbb-3c4a92ee9b5c (disk) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly. Info 2019-01-02 06:18:25
Successfully restored access to volume 503f2cab-3901da70-efbb-3c4a92ee9b5c (disk) following connectivity issues. Info 2019-01-02 06:18:29
Logfile vmkernel.log
2019-01-02T05:18:24.972Z cpu0:33061)NMP: nmp_ThrottleLogForDevice:3303: Cmd 0x89 (0x43b98023e500, 32790) to dev "naa.600140537fc0a26dc759d4634d91b5d7" on path "vmhba37:C0:T0:L0" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2019-01-02T05:18:24.972Z cpu0:33061)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.600140537fc0a26dc759d4634d91b5d7" state in doubt; requested fast path state update...
2019-01-02T05:18:24.972Z cpu0:33061)ScsiDeviceIO: 2652: Cmd(0x43b98023e500) 0x89, CmdSN 0x2183d from world 32790 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2019-01-02T05:18:25.972Z cpu9:32827)NMP: nmp_ThrottleLogForDevice:3303: Cmd 0x2a (0x43b9802998c0, 137918) to dev "naa.600140537fc0a26dc759d4634d91b5d7" on path "vmhba37:C0:T0:L0" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2019-01-02T05:18:25.972Z cpu9:32827)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.600140537fc0a26dc759d4634d91b5d7" state in doubt; requested fast path state update...
2019-01-02T05:18:25.972Z cpu9:32827)ScsiDeviceIO: 2595: Cmd(0x43b9802998c0) 0x2a, CmdSN 0x800e000f from world 137918 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:25.974Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b985c46a00) 0x2a, CmdSN 0x800e000d from world 137918 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:25.978Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b985c3d8c0) 0x2a, CmdSN 0x800e002d from world 137918 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:25.981Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b9802a9b80) 0x2a, CmdSN 0x800e0062 from world 137918 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:25.985Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b98598e3c0) 0x2a, CmdSN 0x800e004f from world 137918 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:26.023Z cpu8:32827)NMP: nmp_ThrottleLogForDevice:3236: last error status from device naa.600140537fc0a26dc759d4634d91b5d7 repeated 10 times
2019-01-02T05:18:26.187Z cpu8:32827)NMP: nmp_ThrottleLogForDevice:3236: last error status from device naa.600140537fc0a26dc759d4634d91b5d7 repeated 20 times
2019-01-02T05:18:26.199Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b580253b00) 0x2a, CmdSN 0x800e0075 from world 137937 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:26.203Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b586baa940) 0x2a, CmdSN 0x800e0012 from world 137937 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:26.207Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b58022a380) 0x2a, CmdSN 0x800e0022 from world 137937 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:26.211Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b586424440) 0x2a, CmdSN 0x800e003b from world 137937 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:26.215Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b58024b840) 0x2a, CmdSN 0x800e0061 from world 137937 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:26.428Z cpu8:32827)ScsiDeviceIO: 2595: Cmd(0x43b980258680) 0x89, CmdSN 0x2183e from world 32790 to dev "naa.600140537fc0a26dc759d4634d91b5d7" failed H:0x8 D:0x0 P:0x0
2019-01-02T05:18:26.530Z cpu1:33321)NMP: nmp_ThrottleLogForDevice:3253: last error status from device naa.600140537fc0a26dc759d4634d91b5d7 repeated 28 times
2019-01-02T05:18:29.866Z cpu8:32827)HBX: 283: 'Disk': HB at offset 3301888 - Reclaimed heartbeat [Timeout]:
2019-01-02T05:18:29.866Z cpu8:32827) [HB state abcdef02 offset 3301888 gen 49 stampUS 499242007896 uuid 5c24abbf-49e8aee0-172f-002655d41fb4 jrnl <FB 2329224> drv 14.61 lockImpl 3]
2019-01-02T05:18:29.866Z cpu8:32827)FS3Misc: 1759: Long VMFS rsv time on 'Disk' (held for 3336 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors