Hey Guys,
Got a really odd one here, for the last week I noticed all VM backups (Veeam 9.5) blew out by 35-45mins each. The problem is with 70+ VMs this pushed us outside the backup Window. In terms of troubleshooting so far we haven't got far, I've reviewed the vCenter logs and rebooted EVERYTHING. What we found from this is that when we reboot vCenter we have no issues for about 15 snapshots. The second thing we tried was snapshots from vCenter (rather than Veeam) but we have the same issue and it's only in creation, not removal. No matter how large or small the VM they all take the same time (even a 5GB CentOS Server) roughly 35-45mins. All tha past tasks looking at the vCeneter logs were between 10-130seconds for the
From what i can see in the vCenter logs there is a timeout peroid that is hit causing the task to be canceled/stopped but this is not causing a failure or any issues, everything still works fine. Its just takes 40mins longer per VM for any snapshots.
Here are the the vCenter logs:
2017-04-14T23:07:39.287+10:00 error vpxd[05084] [Originator@6876 sub=MoEvent opID=4569d102] [EventManagerMoImpl::EventBatchAppender::EventAppenderJob::PostJob] VirtualCenter did not respond to the request to append events after 2400 seconds
2017-04-14T23:07:39.291+10:00 error vpxd[05084] [Originator@6876 sub=Vmomi opID=4569d102] [VpxVmomi] IO exception while sending fault
2017-04-14T23:07:39.291+10:00 info vpxd[05084] [Originator@6876 sub=vpxLro opID=4569d102] [VpxLRO] -- FINISH task-internal-27952
2017-04-14T23:07:39.291+10:00 info vpxd[05084] [Originator@6876 sub=Default opID=4569d102] [VpxLRO] -- ERROR task-internal-27952 -- EventManager -- vim.event.EventManager.logUserEvent: vmodl.fault.UnexpectedFault:
--> Result:
--> (vmodl.fault.UnexpectedFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultName = "vim.fault.Timedout",
--> fault = (vmodl.MethodFault) null,
--> msg = ""
--> }
--> Args:
-->
--> Arg entity:
--> 'vim.VirtualMachine:vm-2741'
--> Arg msg:
--> "Source: Veeam Backup
--> Action: Job "Backup"
--> Operation: Stopped
--> Status: Success
--> ID: 9444fd99-6cb7-45c0-90b7-cead7533a244
--> Description: "
2017-04-14T23:07:39.297+10:00 warning vpxd[05316] [Originator@6876 sub=VpxProfiler opID=HB-host-19@28497-1d592c26] VpxdInvtVm::ProcessVmInfo [ProcessVmInfoTime] took 1477703 ms
2017-04-14T23:07:39.297+10:00 warning vpxd[05316] [Originator@6876 sub=VpxProfiler opID=HB-host-19@28497-1d592c26] VpxdInvtVm::ProcessVmChanges [ProcessVmChangesTime] took 1477703 ms
2017-04-14T23:07:39.300+10:00 warning vpxd[05316] [Originator@6876 sub=VpxProfiler opID=HB-host-19@28497-1d592c26] [VpxdHostSync] ProcessChanges host:syd-sy1-ex02.cloud.local [ProcessChanges] took 1477706 ms
2017-04-14T23:07:39.301+10:00 warning vpxd[05316] [Originator@6876 sub=VpxProfiler opID=HB-host-19@28497-1d592c26] [VpxdHostSync] DoHostSync:0000000004A81420 [DoHostSyncTime] took 1478249 ms
2017-04-14T23:07:39.301+10:00 warning vpxd[05316] [Originator@6876 sub=VpxProfiler opID=HB-host-19@28497-1d592c26] InvtHostSyncLRO::StartWork [HostSyncTime] took 1478249 ms
2017-04-14T23:07:39.301+10:00 warning vpxd[05316] [Originator@6876 sub=VpxProfiler opID=HB-host-19@28497-1d592c26] VpxLro::LroMain [TotalTime] took 1478249 ms
Going bonkers here guys, not idea whats going on. Thanks in advance for any help
I've created a project on Freelancer also:vmware vcenter | VMware
Cheer.