We've been seeing ESXi hosts disconnect from vCenter for approximately 2-12 minutes. After this period, the host reconnects back to the vCetner server. This happens about 2-3 times per 24 hours. DNS and general networking seem fine. From vpxd's perspective, we see timeouts due to components not responding during host sync requests.
During this time period, we see some interesting log messages in vpxa.log:
We see 2-5 of the following messages. This message appears at the begining of the event, and is also interspursped throughout the event:
"2012-02-02T18:36:18.466Z [42931B90 verbose 'Default' opID=HB-host-41@10260-92e04dfd-ab] [VpxaMoService::GetChangesInt] Vpxa restarted or stolen by other server. Start a full sync
2012-02-02T18:40:48.518Z [429E8B90 verbose 'Default' opID=HB-host-41@10332-a9b72171-ce] [VpxaMoService::GetChangesInt] Vpxa restarted or stolen by other server. Start a full sync"
We also see these messages in Vpxa during the disconnect event:
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:21.352Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:41.358Z [428CEB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:41.358Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:41.358Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:41.358Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:41.358Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:41.358Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:36:41.358Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:01.355Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:01.355Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:01.355Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:01.355Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:01.355Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:01.355Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:01.355Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:21.355Z [FFC78780 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:21.355Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:21.355Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:21.356Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:21.356Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:21.356Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:41.349Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:41.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:41.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:41.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:41.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:41.350Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:37:41.350Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:01.350Z [42994B90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:01.350Z [42994B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:01.350Z [42994B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:01.350Z [42994B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:01.350Z [42994B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:01.350Z [42994B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:01.350Z [42994B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:21.349Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:21.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:21.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:21.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:21.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:21.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:21.349Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:41.351Z [FFC78780 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:41.351Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:41.351Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:41.351Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:41.351Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:41.351Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:38:41.351Z [FFC78780 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:01.352Z [FFD1EB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:01.352Z [FFD1EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:01.352Z [FFD1EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:01.352Z [FFD1EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:01.352Z [FFD1EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:01.352Z [FFD1EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:01.352Z [FFD1EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:21.353Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:21.353Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:21.353Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:21.353Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:21.353Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:21.353Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:21.353Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:41.359Z [428CEB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:41.359Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:41.359Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:41.359Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:41.359Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:41.359Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:39:41.359Z [428CEB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:01.360Z [42A8EB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:01.360Z [42A8EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:01.360Z [42A8EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:01.360Z [42A8EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:01.360Z [42A8EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:01.360Z [42A8EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:01.360Z [42A8EB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:21.357Z [42A09B90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:21.357Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:21.357Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:21.357Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:21.357Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:21.357Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:21.357Z [42A09B90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:41.358Z [42A6DB90 warning 'Default'] [FetchQuickStats] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:41.359Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:41.359Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:41.359Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:41.359Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:41.359Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
2012-02-02T18:40:41.359Z [42A6DB90 warning 'Default'] [AddEntityMetric] GetTranslators -- host to vpxd translation is empty. Dropping results
So far, we have found no suspicious messages in the hostd or vmkernel logs.
Reaching out to see if anyone had anymore information on these messages and if anyone had seen this behaviour before.
Regards,
Lukas