I am using ESXi 6.5 and having trouble using PCI passthrough of four GPUs to a single Windows 10 guest VM. They seem to pass through OK initially, before Windows has installed the driver for them. I can see the four GPUs in Device Manager with the little yellow indicators for the drivers not being installed. Once the drivers are installed and Windows starts recognizing them, it crashes. Subsequent startups will crash right after displaying the initial Windows splash screen and before getting to the login screen. Removing all GPUs except for one will work, and the crashing stops. I have tried each GPU individually and they all work with no crashing. I can pass one GPU each to two individual VMs and run them at the same time with no problems. The crashing begins as soon as I try to pass a second GPU to one virtual machine. Bypassing ESXi entirely and using Windows 10 as the main OS I am able to use all four GPUs, so I think perhaps I have an incorrect setting.
My hardware is:
Supermicro X10SRL-F motherboard, Intel Xeon 2620 V4 CPU, 32 GB RAM, 4x Radeon RX580 GPUs.
I have enabled above 4G encoding and configured my motherboard BIOS settings according to this article from Nvidia (It's not Radeon but I figured it would apply to my application as well).
MMIOHBase = 2T
MMIO High Size = 256G
All PCIe oprom = EFI
Onboard LAN oprom type = EFI
I have also added the following lines to my Windows 10 VM configuration:
firmware=“efi"
pciPassthru.use64bitMMIO="TRUE"
according to the KB article I found here:
VMware vSphere VMDirectPath I/O: Requirements for Platforms and Devices (2142307) | VMware KB
Does anyone have an idea of what might be happening here?