Random VMs at what seems to be random times stop being able to communicate with each other.
Example
VM1 on host 1
VM2 on host 2
VM1 cannot ping VM2
VM1 sees an ARP entry in it's ARP cache for VM2
VM2 does NOT see an APR entry in cache
VM1 is able to ping other VMs on the same network
VM2 is able to ping other VMs on the same network
VM1 is able to ping VM2 if they are on the same host
Cleared ARP cache on VM1 and it is removed successfully. Pinged VM again, but no response, however ARP is back in the cache as expected
To confirm, it is not just ping, it is a range of ports that is not working which should be.
No firewalls in the mix either
Also worth noting that this happens when the VMs are on different blades within the same chassis or on different chassis
Versions
ESXi & vCenter is 6.5 Update 1
Compute hardware is Dell FX2 Chassis with FC430 blades and dual FN2210 IO Modules
This happened after migrating VMs from older 5.5 Update 3 ESXi rack mount servers
HW versions and VMware tools have not yet been upgraded (not tested)
Any advise is appreciated