I am trying to re-add a disconnected cluster. There was an issue with the license, which has been recently resolved and the new license has been applied.
I have had a quick look through the existing threads and cannot find a comparable issue.
Things of note:
Specifically the vDRS has been turned off on the cluster, and at the moment it is not required (single host).
The IP address of the vcenter has also changed, it is very likely either of the license change or the recent IP address change have caused the issue (it needed to be moved to another network).
I have updated the vpxa.conf with the new IP address, and have set the port to no avail.
vCenter host has also been rebooted.
vpxa.log does not seem to be updating.
vCenter Version: 6.50.20000 Build 9451637
6.5.0 Build Number 8294253
There is a firewall between the 2 however the firewall is definitely not blocking traffic between the two IP addresses.
Here is the output of the syslog:
2018-12-11T15:57:40Z watchdog-vpxa: [8456549] Begin '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa', min-uptime = 60, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = '', reboot-flag = '0'
2018-12-11T15:57:40Z watchdog-vpxa: Executing '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa'
2018-12-11T15:57:45Z backup.sh.8456580: Locking esx.conf
2018-12-11T15:57:45Z backup.sh.8456580: Creating archive
2018-12-11T15:57:45Z backup.sh.8456580: Unlocking esx.conf
2018-12-11T15:57:50Z watchdog-vpxa: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa' exited after 10 seconds (quick failure 1) 134
2018-12-11T15:57:50Z watchdog-vpxa: Executing '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa'
2018-12-11T15:57:54Z smad[8063535]: Failed: no available HP iLO channels
2018-12-11T15:57:54Z smad[8063535]: Failed bmc_init(). Exit smad.
2018-12-11T15:58:01Z watchdog-vpxa: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa' exited after 11 seconds (quick failure 2) 134
2018-12-11T15:58:01Z watchdog-vpxa: Executing '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa'
2018-12-11T15:58:09Z smad[100962]: Failed: no available HP iLO channels
2018-12-11T15:58:09Z smad[100962]: Failed bmc_init(). Exit smad.
2018-12-11T15:58:12Z watchdog-vpxa: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa' exited after 11 seconds (quick failure 3) 134
2018-12-11T15:58:12Z watchdog-vpxa: Executing '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa'
2018-12-11T15:58:23Z watchdog-vpxa: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa' exited after 11 seconds (quick failure 4) 134
2018-12-11T15:58:23Z watchdog-vpxa: Executing '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa'
2018-12-11T15:58:24Z smad[8456832]: Failed: no available HP iLO channels
2018-12-11T15:58:24Z smad[8456832]: Failed bmc_init(). Exit smad.
2018-12-11T15:58:34Z watchdog-vpxa: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa' exited after 11 seconds (quick failure 5) 134
2018-12-11T15:58:34Z watchdog-vpxa: Executing '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa'
2018-12-11T15:58:39Z smad[8456846]: Failed: no available HP iLO channels
2018-12-11T15:58:39Z smad[8456846]: Failed bmc_init(). Exit smad.
2018-12-11T15:58:44Z watchdog-vpxa: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa' exited after 10 seconds (quick failure 6) 134
2018-12-11T15:58:44Z watchdog-vpxa: End '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa', failure limit reached
vpxa.conf (IP addresses obfuscated)
<config>
<dvs>
<portSyncBatchLimit>100</portSyncBatchLimit>
</dvs>
<httpNfc>
<accessMode>proxyAuto</accessMode>
<enabled>true</enabled>
</httpNfc>
<level id="SoapAdapter.HTTPService">
<logLevel>info</logLevel>
<logName>SoapAdapter.HTTPService</logName>
</level>
<level id="SoapAdapter.HTTPService.HttpConnection">
<logLevel>info</logLevel>
<logName>SoapAdapter.HTTPService.HttpConnection</logName>
</level>
<log>
<level>verbose</level>
<maxFileNum>10</maxFileNum>
<maxFileSize>1048576</maxFileSize>
<memoryLevel>verbose</memoryLevel>
<outputToConsole>false</outputToConsole>
<outputToFiles>false</outputToFiles>
<outputToSyslog>true</outputToSyslog>
<syslog>
<facility>local4</facility>
<ident>Vpxa</ident>
<logHeaderFile>/var/run/vmware/vpxaLogHeader.txt</logHeaderFile>
</syslog>
</log>
<nfc>
<loglevel>error</loglevel>
</nfc>
<task>
<completedMaxEntries>1000</completedMaxEntries>
<maxThreads>98</maxThreads>
<minCompletedLifetime>120</minCompletedLifetime>
</task>
<trace>
<mutex>
<profiledMutexes>InvtLock</profiledMutexes>
</mutex>
<vmomi>
<calls>false</calls>
</vmomi>
</trace>
<vmacore>
<http>
<defaultClientPoolConnectionsPerServer>300</defaultClientPoolConnectionsPerServer>
</http>
<soap>
<sessionTimeout>1440</sessionTimeout>
</soap>
<ssl>
<doVersionCheck>false</doVersionCheck>
</ssl>
<threadPool>
<IoMax>9</IoMax>
<TaskMax>4</TaskMax>
<ThreadStackSizeKb>128</ThreadStackSizeKb>
<threadNamePrefix>vpxa</threadNamePrefix>
</threadPool>
</vmacore>
<vpxa>
<bundleVersion>1000000</bundleVersion>
<datastorePrincipal>root</datastorePrincipal>
<hostIp>X.X.X.X</hostIp>
<hostKey/>
<hostPort>443</hostPort>
<licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
<memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
<serverIp>Y.Y.Y.Y</serverIP>
<serverPort>902</serverPort>
</vpxa>
<workingDir>/var/log/vmware/vpx</workingDir>
Update:
vmkernel.log:
2018-12-11T20:58:11.027Z cpu13:8510156)User: 3089: vpxa: wantCoreDump:vpxa signal:6 exitCode:0 coredump:enabled
2018-12-11T20:58:11.203Z cpu4:8510156)UserDump: 3024: vpxa: Dumping cartel 8510156 (from world 8510156) to file /var/core/vpxa-zdump.003 ...
2018-12-11T20:58:21.532Z cpu16:8510156)UserDump: 3172: vpxa: Userworld(vpxa) coredump complete.