Hello, I have a question for Vmotion a virtual machine failed.
The vmvare.log reported: Failed to resume on the destination: Failure. Exiting...
The vmkernel.log :
2017-07-20T07:19:24.315Z cpu12:4091250)Migrate: vm 4091262: 3382: Setting FSR info: Dest ts = 1500506207908405, src ip = <127.0.0.1> dest ip = <127.0.0.1> Dest wid = 0 using UNSHARED swap
2017-07-20T07:19:24.317Z cpu12:4091250)WARNING: Migrate: 3451: 1500506207908405 D: Unable to set NUMA affinity mask for world 4091262
2017-07-20T07:19:24.317Z cpu12:4091250)Hbr: 3394: Migration start received (worldID=4091262) (migrateType=2) (event=0) (isSource=0) (sharedConfig=0)
2017-07-20T07:19:24.395Z cpu4:4090868)Migrate: vm 4090869: 3382: Setting FSR info: Source ts = 1500506207908405, src ip = <127.0.0.1> dest ip = <127.0.0.1> Dest wid = 4091262 using UNSHARED swap
2017-07-20T07:19:24.395Z cpu4:4090868)Hbr: 3394: Migration start received (worldID=4090869) (migrateType=2) (event=0) (isSource=1) (sharedConfig=0)
2017-07-20T07:19:24.823Z cpu14:4091265)Swap: vm 4091262: 1419: Successfully initialized normal swapfile.
2017-07-20T07:19:24.823Z cpu14:4091265)FSR: 1500: 1500506207908405 S: Swap copy helper world starting swap copy.
2017-07-20T07:19:24.823Z cpu14:4091265)SVM: 5041: SkipZero 0, dstFsBlockSize 0, preallocateBlocks 0, vmfsOptimizations 1, useBitmapCopy 0, skipPlugGrain 0
2017-07-20T07:19:24.823Z cpu14:4091265)SVM: 5139: SVM_MakeDev.5139: Creating device 1d2ca53-1aecc89-svmmirror: Success
2017-07-20T07:19:24.823Z cpu14:4091265)SVM: 5188: Created device 1d2ca53-1aecc89-svmmirror, primary 1d2ca53, secondary 1aecc89
2017-07-20T07:19:24.826Z cpu2:4091266)SVM: 2858: scsi255:255 Completed copy in 3 ms. vmmLeaderID = 4090869.
2017-07-20T07:19:24.826Z cpu2:4091266)FSR: 1186: 1500506207908405 S: Swap copy complete successfully.
2017-07-20T07:19:24.826Z cpu14:4091265)FSR: 1534: 1500506207908405 S: Swap copy complete.
2017-07-20T07:19:26.765Z cpu15:33056)ScsiDeviceIO: 2646: Cmd(0x439d808c6d00) 0x1a, CmdSN 0x582c57 from world 0 to dev "naa.690b11c00f2a300020bc0d721364929d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2017-07-20T07:19:27.097Z cpu5:4090869)NetPort: 1780: disabled port 0x20102e5
2017-07-20T07:19:27.097Z cpu5:4090874)VSCSI: 6699: handle 69883(vscsi0:0):Destroying Device for world 4090869 (pendCom 0)
2017-07-20T07:19:27.100Z cpu6:4090871)SVM: 5041: SkipZero 0, dstFsBlockSize 0, preallocateBlocks 0, vmfsOptimizations 1, useBitmapCopy 0, skipPlugGrain 0
2017-07-20T07:19:27.100Z cpu6:4090871)SVM: 5139: SVM_MakeDev.5139: Creating device 1deca69-29ccc8d-svmmirror: Success
2017-07-20T07:19:27.100Z cpu6:4090871)SVM: 5188: Created device 1deca69-29ccc8d-svmmirror, primary 1deca69, secondary 29ccc8d
2017-07-20T07:19:27.101Z cpu5:4090869)NetPort: 1573: enabled port 0x20102e5 with mac 00:50:56:8c:57:9a
2017-07-20T07:19:27.107Z cpu5:4090874)SVM: 5041: SkipZero 1, dstFsBlockSize 1048576, preallocateBlocks 1, vmfsOptimizations 1, useBitmapCopy 0, skipPlugGrain 0
2017-07-20T07:19:27.107Z cpu5:4090874)SVM: 5139: SVM_MakeDev.5139: Creating device 208ca6c-1ffcc8a-svmmirror: Success
2017-07-20T07:19:27.107Z cpu5:4090874)SVM: 5188: Created device 208ca6c-1ffcc8a-svmmirror, primary 208ca6c, secondary 1ffcc8a
2017-07-20T07:19:27.108Z cpu5:4090874)VSCSI: 4011: handle 69885(vscsi0:0):Creating Virtual Device for world 4090869 (FSS handle 38652525) numBlocks=83886080 (bs=512)
2017-07-20T07:19:27.108Z cpu5:4090874)VSCSI: 273: handle 69885(vscsi0:0):Input values: res=0 limit=-1 bw=-1 Shares=-1
2017-07-20T07:19:27.206Z cpu9:4091271)SVM: 2858: scsi0:0 Completed copy in 89 ms. vmmLeaderID = 4090869.
2017-07-20T07:19:27.290Z cpu12:2674309)ScsiDeviceIO: 2646: Cmd(0x439d809540c0) 0x1a, CmdSN 0x582c8a from world 0 to dev "naa.690b11c00f2a300020bc0d721364929d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2017-07-20T07:19:27.291Z cpu12:2674309)ScsiDeviceIO: 2646: Cmd(0x439d809540c0) 0x1a, CmdSN 0x582c8f from world 0 to dev "naa.690b11c00f2a300020bc0d721364929d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2017-07-20T07:19:27.298Z cpu1:4091272)SVM: 2851: file# 0: Completed copy in 80 ms. vmmLeaderID = 4090869.
2017-07-20T07:19:27.305Z cpu1:4090869)NetPort: 1780: disabled port 0x20102e5
2017-07-20T07:19:27.305Z cpu1:4090874)VSCSI: 6699: handle 69885(vscsi0:0):Destroying Device for world 4090869 (pendCom 0)
2017-07-20T07:19:27.306Z cpu1:4090874)SVM: 2428: SVM Mirrored mode IO stats for device: 208ca6c-1ffcc8a-svmmirror
2017-07-20T07:19:27.306Z cpu1:4090874)SVM: 2432: Total # IOs mirrored: 0, Total # IOs sent only to source: 0, Total # IO deferred by lock: 0
2017-07-20T07:19:27.306Z cpu1:4090874)SVM: 2436: Deferred IO stats - Max: 0, Total: 0, Avg: 1 (msec)
2017-07-20T07:19:27.306Z cpu1:4090874)SVM: 2450: Destroyed device 208ca6c-1ffcc8a-svmmirror
2017-07-20T07:19:27.308Z cpu1:4090874)SVM: 2428: SVM Mirrored mode IO stats for device: 1deca69-29ccc8d-svmmirror
2017-07-20T07:19:27.308Z cpu1:4090874)SVM: 2432: Total # IOs mirrored: 0, Total # IOs sent only to source: 0, Total # IO deferred by lock: 0
2017-07-20T07:19:27.308Z cpu1:4090874)SVM: 2436: Deferred IO stats - Max: 0, Total: 0, Avg: 1 (msec)
2017-07-20T07:19:27.308Z cpu1:4090874)SVM: 2450: Destroyed device 1deca69-29ccc8d-svmmirror
2017-07-20T07:19:27.529Z cpu0:4091250)Swap: vm 4091262: 1289: Swapfile already initialized. No work to do.
2017-07-20T07:19:27.574Z cpu3:4091250)VSCSI: 4011: handle 69886(vscsi0:0):Creating Virtual Device for world 4091262 (FSS handle 44092557) numBlocks=83886080 (bs=512)
2017-07-20T07:19:27.574Z cpu3:4091250)VSCSI: 273: handle 69886(vscsi0:0):Input values: res=0 limit=-2 bw=-1 Shares=1000
2017-07-20T07:19:27.642Z cpu4:3265078)ScsiDeviceIO: 2646: Cmd(0x439d84839ec0) 0x1a, CmdSN 0x582d21 from world 0 to dev "naa.690b11c00f2a300020bc0d721364929d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2017-07-20T07:19:27.643Z cpu9:4091262)VMMVMKCall: 235: Received INIT from world 4091262
2017-07-20T07:19:27.643Z cpu4:3265078)ScsiDeviceIO: 2646: Cmd(0x439d84839ec0) 0x1a, CmdSN 0x582d26 from world 0 to dev "naa.690b11c00f2a300020bc0d721364929d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2017-07-20T07:19:27.650Z cpu1:34778)LVM: 10060: Device naa.600b342b1d453c8db63dd6856d0000db:1 detected to be a snapshot:
2017-07-20T07:19:27.650Z cpu1:34778)LVM: 10067: queried disk ID: <type 2, len 22, lun 11, devType 0, scsi 0, h(id) 8686437659640839581>
2017-07-20T07:19:27.650Z cpu1:34778)LVM: 10074: on-disk disk ID: <type 2, len 22, lun 0, devType 0, scsi 0, h(id) 9512531387408291070>
2017-07-20T07:19:27.653Z cpu1:34778)LVM: 10060: Device naa.600b3429a5bbbbfdfbb1d10c6d0000db:1 detected to be a snapshot:
2017-07-20T07:19:27.653Z cpu1:34778)LVM: 10067: queried disk ID: <type 2, len 22, lun 12, devType 0, scsi 0, h(id) 3064839419528190981>
2017-07-20T07:19:27.653Z cpu1:34778)LVM: 10074: on-disk disk ID: <type 2, len 22, lun 0, devType 0, scsi 0, h(id) 9512531387408291070>
2017-07-20T07:19:27.655Z cpu9:4091265)SVM: 2463: Error destroying device 1d2ca53-1aecc89-svmmirror (Busy)
2017-07-20T07:19:27.655Z cpu9:4091265)WARNING: FSR: 1269: 1500506207908405 S: Failed to destroy mirror node for swap file: Busy
2017-07-20T07:19:27.655Z cpu9:4091265)WARNING: FSR: 1589: 1500506207908405 S: Swap copy helper failed, exiting.
2017-07-20T07:19:27.655Z cpu9:4091265)WARNING: Migrate: 270: 1500506207908405 S: Failed: Busy (0xbad0004) @0x418036c1231e
2017-07-20T07:19:27.656Z cpu0:4091262)WARNING: FSR: 4726: 1500506207908405 D: Failed to tranfer the swap file from source VM to destination VM.
2017-07-20T07:19:27.656Z cpu0:4091262)WARNING: Migrate: 270: 1500506207908405 D: Failed: Failure (0xbad0001) @0x418037cc01d6
2017-07-20T07:19:27.666Z cpu4:4090868)WARNING: Migrate: 5451: 1500506207908405 S: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error.
2017-07-20T07:19:27.680Z cpu1:4091275)WARNING: Migrate: 5451: 1500506207908405 D: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error.
And then, I could shutdown this vm, but it couldn't be poweron again. The vmkernel.log reported that:
2017-07-20T09:25:52.430Z cpu2:4172955)FSS: 6146: Failed to open file 'zmx_hot-3-d0fd9b24.vswp'; Requested flags 0x8, world: 4172955 [vmx], (Existing flags 0x8, world: 4090868 [unknown]): Busy
2017-07-20T09:25:52.430Z cpu2:4172955)WARNING: SwapFileOps: vm 4172956: 286: Failed to open swap file '/vmfs/volumes/5965f31b-a9168818-fc27-90b11c095e3c/zmx_hot-3_3/zmx_hot-3-d0fd9b24.vswp': Busy
2017-07-20T09:25:52.430Z cpu2:4172955)WARNING: Swap: 3547: Failed to initialize swap file '/vmfs/volumes/5965f31b-a9168818-fc27-90b11c095e3c/zmx_hot-3_3/zmx_hot-3-d0fd9b24.vswp' : Busy