Hi,
I am trying to enable SR-IOV on my server.
I have a Dell R630 running the evaluation version of ESXi 6.0u3
# esxcli system version get Product: VMware ESXi Version: 6.0.0 Build: Releasebuild-5572656 Update: 3 Patch: 69
The two network cards are Intel I350-t (quad ports and dual ports) :
# esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:01:00.0 igb Up 1000Mbps Full 24:6e:96:5f:aa:88 1500 Intel Corporation I350 Gigabit Network Connection
vmnic1 0000:01:00.1 igb Up 1000Mbps Full 24:6e:96:5f:aa:89 9000 Intel Corporation I350 Gigabit Network Connection
vmnic2 0000:01:00.2 igb Up 1000Mbps Full 24:6e:96:5f:aa:8a 1500 Intel Corporation I350 Gigabit Network Connection
vmnic3 0000:01:00.3 igb Up 1000Mbps Full 24:6e:96:5f:aa:8b 1500 Intel Corporation I350 Gigabit Network Connection
vmnic4 0000:04:00.0 igb Up 1000Mbps Full a0:36:9f:e7:6a:e6 1500 Intel Corporation I350 Gigabit Network Connection
vmnic5 0000:04:00.1 igb Up 1000Mbps Full a0:36:9f:e7:6a:e7 9000 Intel Corporation I350 Gigabit Network Connection
SR-IOV is enabled in the bios (globally, and for each devices). SR-IOV is marked as supported and disabled in vCenter.
I don't have a max_vfs parameter for the igb driver :
# esxcli system module parameters list -m igb
Name Type Value Description
--------------------- ------------ ----- ----------------------------------------------------------------------------------------------------
DMAC array of int Disable or set latency for DMA Coalescing ((0=off, 1000-10000(msec), 250, 500 (usec))
EEE array of int Enable/disable on parts that support the feature
IntMode array of int Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2
InterruptThrottleRate array of int Maximum interrupts per second, per vector, (max 100000), default 3=adaptive
LLIPort array of int Low Latency Interrupt TCP Port (0-65535), default 0=off
LLIPush array of int Low Latency Interrupt on TCP Push flag (0,1), default 0=off
LLISize array of int Low Latency Interrupt on Packet Size (0-1500), default 0=off
MDD array of int Malicious Driver Detection (0/1), default 1 = enabled. Only available when max_vfs is greater than 0
Node array of int set the starting node to allocate memory on, default -1
QueuePairs array of int Enable Tx/Rx queue pairs for interrupt handling (0,1), default 1=on
RSS array of int Number of Receive-Side Scaling Descriptor Queues (0-8), default 1, 0=number of cpus
VMDQ array of int Number of Virtual Machine Device Queues: 0-1 = disable, 2-8 enable, default 0
debug int Debug level (0=none, ..., 16=all)
heap_initial int Initial heap size allocated for the driver.
heap_max int Maximum attainable heap size for the driver.
skb_mpool_initial int Driver's minimum private socket buffer memory pool size.
skb_mpool_max int Maximum attainable private socket buffer memory pool size for the driver.
If I try the esxcli command as described here, I have the following error :
# esxcli system module parameters set -m igb -p "max_vfs=0,1,0,0,1,0"
Unable to set module parameters the following params are invalid: max_vfs
The content of hostd.log :
2017-08-16T07:57:54.896Z error hostd[6AEC1B70] [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-80-c95c user=root] GetPrimitiveParam: Cannot find (help)
2017-08-16T07:57:54.896Z info hostd[6AEC1B70] [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-80-c95c user=root] Dispatch set
2017-08-16T07:57:54.897Z info hostd[6AEC1B70] [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-80-c95c user=root] Dispatch set done
2017-08-16T07:57:54.897Z info hostd[6AEC1B70] [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-80-c95c user=root] Dispatch set failed
2017-08-16T07:57:54.897Z info hostd[6AEC1B70] [Originator@6876 sub=Default opID=esxcli-80-c95c user=root] AdapterServer caught exception: vim.EsxCLI.CLIFault
2017-08-16T07:57:54.897Z info hostd[6AEC1B70] [Originator@6876 sub=Solo.Vmomi opID=esxcli-80-c95c user=root] Activation [N5Vmomi10ActivationE:0x6b1295d0] : Invoke done [set] on [vim.EsxCLI.system.module.parameters:ha-cli-handler-system-module-parameters]
2017-08-16T07:57:54.897Z verbose hostd[6AEC1B70] [Originator@6876 sub=Solo.Vmomi opID=esxcli-80-c95c user=root] Arg append:
--> (null)
2017-08-16T07:57:54.897Z verbose hostd[6AEC1B70] [Originator@6876 sub=Solo.Vmomi opID=esxcli-80-c95c user=root] Arg force:
--> (null)
2017-08-16T07:57:54.897Z verbose hostd[6AEC1B70] [Originator@6876 sub=Solo.Vmomi opID=esxcli-80-c95c user=root] Arg module:
--> "igb"
2017-08-16T07:57:54.897Z verbose hostd[6AEC1B70] [Originator@6876 sub=Solo.Vmomi opID=esxcli-80-c95c user=root] Arg parameterstring:
--> "max_vfs=0,1,0,0,1,0"
2017-08-16T07:57:54.897Z info hostd[6AEC1B70] [Originator@6876 sub=Solo.Vmomi opID=esxcli-80-c95c user=root] Throw vim.EsxCLI.CLIFault
2017-08-16T07:57:54.897Z info hostd[6AEC1B70] [Originator@6876 sub=Solo.Vmomi opID=esxcli-80-c95c user=root] Result:
--> (vim.EsxCLI.CLIFault) {
--> faultCause = (vmodl.MethodFault) null,
--> errMsg = (string) [
--> "Unable to set module parameters the following params are invalid: max_vfs "
--> ],
--> msg = ""
--> }
2017-08-16T07:57:54.897Z warning hostd[6AEC1B70] [Originator@6876 sub=Locale opID=esxcli-80-c95c user=root] Resource module 'EsxCLI' not found.
2017-08-16T07:57:54.897Z warning hostd[6AEC1B70] [Originator@6876 sub=Locale opID=esxcli-80-c95c user=root] Resource module 'EsxCLI' not found.
When I try with the vSphere web client :
2017-08-16T08:03:57.907Z info hostd[69C80B70] [Originator@6876 sub=Hostsvc opID=3a8c2c12-a42b-4adf-822a-45e9a8818bbf-4287-ngc-f9-1b-c9fe user=vpxuser:xxx] Modifying SR-IOV Configuration of [igb]
2017-08-16T08:03:57.907Z error hostd[69C80B70] [Originator@6876 sub=Hostsvc opID=3a8c2c12-a42b-4adf-822a-45e9a8818bbf-4287-ngc-f9-1b-c9fe user=vpxuser:xxx] Failed to update SR-IOV configuration of module: igb
2017-08-16T08:03:57.908Z info hostd[69C80B70] [Originator@6876 sub=Default opID=3a8c2c12-a42b-4adf-822a-45e9a8818bbf-4287-ngc-f9-1b-c9fe user=vpxuser:xxx] AdapterServer caught exception: vim.fault.PlatformConfigFault
2017-08-16T08:03:57.908Z info hostd[69C80B70] [Originator@6876 sub=Vimsvc.TaskManager opID=3a8c2c12-a42b-4adf-822a-45e9a8818bbf-4287-ngc-f9-1b-c9fe user=vpxuser:xxx] Task Completed : haTask-ha-host-vim.host.PciPassthruSystem.updatePassthruConfig-66872711 Status error
2017-08-16T08:03:57.908Z info hostd[69C80B70] [Originator@6876 sub=Solo.Vmomi opID=3a8c2c12-a42b-4adf-822a-45e9a8818bbf-4287-ngc-f9-1b-c9fe user=vpxuser:xxx] Activation [N5Vmomi10ActivationE:0x69f22c00] : Invoke done [updatePassthruConfig] on [vim.host.PciPassthruSystem:ha-pcipassthrusystem]
2017-08-16T08:03:57.908Z verbose hostd[69C80B70] [Originator@6876 sub=Solo.Vmomi opID=3a8c2c12-a42b-4adf-822a-45e9a8818bbf-4287-ngc-f9-1b-c9fe user=vpxuser:xxx] Arg config:
--> (vim.host.PciPassthruConfig) [
--> (vim.host.SriovConfig) {
--> id = "0000:01:00.0",
--> passthruEnabled = false,
--> sriovEnabled = true,
--> numVirtualFunction = 1
--> }
--> ]
2017-08-16T08:03:57.908Z info hostd[69C80B70] [Originator@6876 sub=Solo.Vmomi opID=3a8c2c12-a42b-4adf-822a-45e9a8818bbf-4287-ngc-f9-1b-c9fe user=vpxuser:xxx] Throw vim.fault.PlatformConfigFault
2017-08-16T08:03:57.908Z info hostd[69C80B70] [Originator@6876 sub=Solo.Vmomi opID=3a8c2c12-a42b-4adf-822a-45e9a8818bbf-4287-ngc-f9-1b-c9fe user=vpxuser:xxx] Result:
--> (vim.fault.PlatformConfigFault) {
--> faultCause = (vmodl.MethodFault) null,
--> text = "Failed to update SR-IOV configuration of module(s): igb",
--> msg = ""
--> }
2017-08-16T08:03:58.000Z info hostd[FFA0CB20] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
--> key = -1,
--> chainId = 1952804399,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> name = "host.tld",
--> host = 'vim.HostSystem:ha-host'
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.net.vmnic.linkstate.down",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> key = "1",
--> value = "vmnic0"
--> }
--> ],
--> objectId = "ha-eventmgr",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null
--> }
2017-08-16T08:03:58.000Z info hostd[FFA0CB20] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 2574 : Physical NIC vmnic0 linkstate is down.
I tried multiple drivers version :
- igb 5.0.5.1
- igb 5.3.3
- igbn 1.3.1
And firmware versions :
- firmware-version: 1.67, 0x80000d38, 17.5.10
- firmware-version: 1.67, 0x80000d38, 18.0.16
I don't know what to try or where to look to go further...
Any idea ?
Thanks in advance !
