Yesterday we had a power issue which caused one of our 4 hosts to stop because of faulty redundant power supplies, but the other 3 hosts was fine,
After fixing power issue and starting the 4th host, some thing strange happens,
We was not able to login to vCenter and it seems to had corrupted data base, there for we've connected to each host directly using vsphere client and there strange thing was every VM that we had was on at least two hosts, and some of them ( specially Windows ones ) had got file system issues.
After that I thought that's maybe because of crashed vCenter database that make it to do something crazy like powering on a VM on two different hosts simultaneously therefor I've powered off the vCenter,
Powering off the vCenter helped us to have time to recover some VMs from backups but after a while I saw an other VM which was fine had become available on two hosts and it was running on both hosts for short time, that VM also got issue in file system but luckily Windows was able to fix it using check disk process.
My question is was caused this none-sense and how can I prevent it from happening again? it really make us nervous feeling that every second an other VM of ours may have experience the same issue.