This happened to me a few months ago on a fresh install of the vcenter appliance 6.5. It just stopped working a week or two after applying an update. Services would not start and there was no indication as to why. It wasn't a space issue, it wasn't that other issue with a duplicate value in the vpostgres database I read about either. I finally gave up and wiped it out to redeploy from scratch.
Well low and behold, sometime last night vcenter stopped working again. This time it wasn't even a full week after having applied the 6.5.0c patch. Only two services start, none of the others will. My deployment is two appliances, a PSC and the vCenter. The PSC appears fine and the services are showing healthy. The vCenter turned to garbage again. Here's an output of service-control:
root@mp1vsivcs501 [ ~ ]# service-control --status Running: lwsmd vmafdd Stopped: applmgmt vmcam vmonapi vmware-cm vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Trying to start any service produces a similar output:
root@mp1vsivcs501 [ ~ ]# service-control --start vmware-vpxd-svcs
Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd-svcs'], include_coreossvcs=False, include_leafossvcs=False
2017-04-24T19:36:49.136Z Running command: ['/usr/bin/systemctl', 'set-environment', 'VMON_PROFILE=NONE']
2017-04-24T19:36:49.140Z Done running command
2017-04-24T19:36:49.143Z Running command: ['/usr/bin/systemctl', 'daemon-reload']
2017-04-24T19:36:49.222Z Done running command
2017-04-24T19:36:49.222Z Running command: ['/usr/bin/systemctl', 'set-property', u'vmware-vmon.service', 'MemoryAccounting=true', 'CPUAccounting=true', 'BlockIOAccounting=true']
2017-04-24T19:36:49.227Z Done running command
2017-04-24T19:36:49.231Z RC = 1
Stdout =
Stderr = Failed to execute operation: Unit file is masked
2017-04-24T19:36:49.231Z { "resolution": null, "detail": [ { "args": [ "Stderr: Failed to execute operation: Unit file is masked\n" ], "id": "install.ciscommon.command.errinvoke", "localized": "An error occurred while invoking external command : 'Stderr: Failed to execute operation: Unit file is masked\n'", "translatable": "An error occurred while invoking external command : '%(0)s'" } ], "componentKey": null, "problemId": null
}
2017-04-24T19:36:49.231Z Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE']
2017-04-24T19:36:49.235Z Done running command
Error executing start on service vpxd-svcs. Details { "resolution": null, "detail": [ { "args": [ "vmware-vmon" ], "id": "install.ciscommon.service.failstart", "localized": "An error occurred while starting service 'vmware-vmon'", "translatable": "An error occurred while starting service '%(0)s'" } ], "componentKey": null, "problemId": null
}
Service-control failed. Error { "resolution": null, "detail": [ { "args": [ "vmware-vmon" ], "id": "install.ciscommon.service.failstart", "localized": "An error occurred while starting service 'vmware-vmon'", "translatable": "An error occurred while starting service '%(0)s'" } ], "componentKey": null, "problemId": null
}
The first thing that pops out for me is line 11, "Failed to execute operation: Unit file is masked". I get that on every service I attempt to start and I'm not finding anything in VMware's knowledge portal about it. This is extremely frustrating.
**Additional info**
Running a search on just unit file is masked took me to a generic ubuntu thread about systemctl showing masked unit files. Here's the output of the systemctl list-unit-files:
root@mp1vsivcs501 [ ~ ]# systemctl list-unit-files | grep vmware vmware-bigsister.service static vmware-cm.service masked vmware-content-library.service masked vmware-eam.service masked vmware-firewall.service enabled vmware-imagebuilder.service masked vmware-mbcs.service masked vmware-netdump.service masked vmware-perfcharts.service masked vmware-rbd-watchdog.service masked vmware-rhttpproxy.service masked vmware-sca.service masked vmware-sps.service masked vmware-statsmonitor.service masked vmware-updatemgr.service masked vmware-vapi.service masked vmware-vcha.service masked vmware-vmon.service masked vmware-vmonapi.service masked vmware-vpostgres.service masked vmware-vpxd-svcs.service masked vmware-vpxd.service masked vmware-vsan-health.service masked vmware-vsm.service masked vmware-bigsister.timer disabled
Not sure if that's normal or not, but it appears to be what the error message is complaining about?
Message was edited by: jhboricua