Hi everyone,
I am the happy owner of six Dell R630 with the free ESXi 6 stantard image (VMware ESXi 6.0.0 build-3620759) installed on SD cards.
The ESXis are split in two different datacenter. There is no shared storage and no vCenter. There are 4 to 12 VMs and plenty of spare resources on each host.
Sometimes, apparently at random, I experience what I can only call an "Unknown:" syslog messages storm from a host or another. Like... millions of them in a few minutes.
The log messages are just rows of :
2017-01-15T20:48:58Z Unknown:
2017-01-15T20:48:58Z Unknown:
2017-01-15T20:48:58Z Unknown:
2017-01-15T20:48:58Z Unknown:
2017-01-15T20:48:58Z Unknown:
2017-01-15T20:48:58Z Unknown:
Enough to blow the 7 logrotate history in less than a minute:
-rw------- 1 root root 2628 Jan 15 20:48 syslog.0.gz
-rw------- 1 root root 2629 Jan 15 20:48 syslog.1.gz
-rw------- 1 root root 2627 Jan 15 20:48 syslog.2.gz
-rw------- 1 root root 2626 Jan 15 20:48 syslog.3.gz
-rw------- 1 root root 2630 Jan 15 20:48 syslog.4.gz
-rw------- 1 root root 2628 Jan 15 20:48 syslog.5.gz
-rw------- 1 root root 2630 Jan 15 20:48 syslog.6.gz
-rw------- 1 root root 2630 Jan 15 20:48 syslog.7.gz
-rw------- 1 root root 420397 Jan 15 21:08 syslog.log
And fill up our 60Go remote syslog server in a few hours (~5 hours, and more or less 405000000 rows, for only one server storming).
Killing the vmsyslogd processes is enough to stop the storm. They are automatically restarted, and i receive normal syslog messages afterward...until the next storm...
I'll dig again in the remote syslog history tomorrow. With a little luck there will be something useful but so far, nothing...
If anyone knows something about my problem or have any clue...help appreciated.
Thanks in advance.