Customer has 5 vSAN ready nodes all Dell PowerEdge 7R525 with AMD EPYC processors.
Setup:
2x S5248F-ON switches with firmware 10.5.1.4.
2x Mellanox ConnectX-5 dual port - firmware 16.27.61.06
ESXi 7.0 - Mellanox driver - Mellanox-nmlx5_4.19.70.1-1OEM.700.1.0.15525992_16253686-package
iperf between hosts we can only get about 15-16Gbps. I wasn't expecting full line rate, but at least 20Gbps+
Host1:
esxcli network firewall set --enabled false
/usr/lib/vmware/vsan/bin/iperf3.copy -s 10.0.0.1
Host2:
esxcli network firewall set --enabled false
/usr/lib/vmware/vsan/bin/iperf3.copy -c 10.0.0.1 -i 1 -t 10 -w 16M
Tuning we have tried, but these make no difference:
esxcfg-advcfg -s 4 /Net/TcpipRxDispatchQueues
esxcfg-advcfg -s 65535 /Net/VmxnetLROMaxLength
esxcli system settings kernel set -s netMaxPktsToProcess -v 128
esxcli system settings kernel set -s intrBalancingEnabled -v false
esxcli network nic ring current set -r 4096 -n vmnic2
esxcli network nic coalesce set -a false -n vmnic2
Any thoughts please, Dell support won't help?