I have a Cisco 3012 blade switch (cbs31x0-universal-mz.122-55.SE5.bin) in an IBM blade chassis with a HS22 blade. The blade is running ESXi 5.5.0, 2068190.
I have a few VMs on this blade already, working no problem. I am trying to add an additional VLAN (4035) to the blade. The issue is that some traffic from the 3012 switch is not hitting the ESXi vswitch. For example STP traffic does is seen by the vswitch, but nothing else (tried ARP, ICMP and Telnet)
Here's the config of the 3012. Gi0/3 is facing the blade. Vlan 4035 is the affected VLAN. To keep things simple, I created an SVI on the 3012 to verify connectivity between it and the vswitch, and kept all tests limited to between the VM and the 3012.
ics01.ht-tor-mow-01#sh run int gi0/3
Building configuration...
Current configuration : 171 bytes
!
interface GigabitEthernet0/3
description monitor blade
switchport trunk allowed vlan 4011,4013,4035,4089
switchport mode trunk
spanning-tree portfast
end
ics01.ht-tor-mow-01#show vlan id 4035
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
4035 VLAN4035 active Gi0/3
VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
4035 enet 104035 1500 - - - - - 0 0
Remote SPAN VLAN
----------------
Disabled
Primary Secondary Type Ports
------- --------- ----------------- ------------------------------------------
ics01.ht-tor-mow-01#sh run int vl4035
Building configuration...
Current configuration : 62 bytes
!
interface Vlan4035
ip address 192.0.2.1 255.255.255.0
end
ics01.ht-tor-mow-01#
If I ping a VM on the same subnet, I see an ARP request generated by the 3012, but I don't see that ARP request hitting the vswitch:
ics01.ht-tor-mow-01#ping 192.0.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.0.2.2, timeout is 2 seconds:
Aug 20 10:35:41.086: IP ARP: sent req src 192.0.2.1 d057.4c0e.92c2,
dst 192.0.2.2 0000.0000.0000 Vlan4035.
Aug 20 10:35:43.091: IP ARP: sent req src 192.0.2.1 d057.4c0e.92c2,
dst 192.0.2.2 0000.0000.0000 Vlan4035.
Aug 20 10:35:45.096: IP ARP: sent req src 192.0.2.1 d057.4c0e.92c2,
dst 192.0.2.2 0000.0000.0000 Vlan4035.
Aug 20 10:35:47.100: IP ARP: sent req src 192.0.2.1 d057.4c0e.92c2,
dst 192.0.2.2 0000.0000.0000 Vlan4035.
Aug 20 10:35:49.105: IP ARP: sent req src 192.0.2.1 d057.4c0e.92c2,
dst 192.0.2.2 0000.0000.0000 Vlan4035.
Success rate is 0 percent (0/5)
ics01.ht-tor-mow-01#
~ # pktcap-uw --uplink vmnic0 --vlan 4035 --stage 0 --dir 0
The name of the uplink is vmnic0
The session filter VLAN is 4035
The Stage is Pre
The dir is Rx
No server port specifed, select 5165 as the port
Output the packet info to console.
Local CID 2
Listen on port 5165
Accept...Vsock connection from port 1050 cid 2
Destroying session 26
Dumped 0 packet to console, dropped 0 packets.
Done.
~ #
The interesting thing, however, is if I enable spanning-tree on VLAN4035, I see those packets hitting the vswitch:
ics01.ht-tor-mow-01#conf t
Enter configuration commands, one per line. End with CNTL/Z.
ics01.ht-tor-mow-01(config)#spanning-tree vlan 4035
ics01.ht-tor-mow-01(config)#
~ # pktcap-uw --uplink vmnic0 --vlan 4035 --stage 0 --dir 0
The name of the uplink is vmnic0
The session filter VLAN is 4035
The Stage is Pre
The dir is Rx
No server port specifed, select 5634 as the port
Output the packet info to console.
Local CID 2
Listen on port 5634
Accept...Vsock connection from port 1051 cid 2
13:54:11.293521[1] Captured at UplinkRcv point, TSO not enabled, Checksum not offloaded and not verified, VLAN tag 4035, length 64.
Segment[0] ---- 64 bytes:
0x0000: 0100 0ccc cccd d057 4c0e 9283 0032 aaaa
0x0010: 0300 000c 010b 0000 0202 0e8f c3d0 574c
0x0020: 0e92 8000 0000 008f c3d0 574c 0e92 8080
0x0030: 0300 0014 0002 000f 0000 0000 0002 0fc3
In the opposite direction, if I ping from the VM to the 3012, the ARP request is processed by the 3012, and a reply is generated, but the reply is not seen by the vswitch:
Verify interface settings on VM:
jlixfeld@monitor:~$ /sbin/ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:0c:29:22:09:e0
inet addr:192.0.2.2 Bcast:192.0.2.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe22:9e0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1188 (1.1 KiB)
Ping the SVI on the 3012 from the VM:
jlixfeld@monitor:~$ ping -c 1 192.0.2.1
PING 192.0.2.1 (192.0.2.1) 56(84) bytes of data.
From 192.0.2.2 icmp_seq=1 Destination Host Unreachable
--- 192.0.2.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
jlixfeld@monitor:~$
A wireshark done on the VM at the same time sees the ARP request being generated:
10:39:57.899289 ARP, Request who-has 192.0.2.1 tell 192.0.2.2, length 28
10:39:58.896637 ARP, Request who-has 192.0.2.1 tell 192.0.2.2, length 28
10:39:59.896635 ARP, Request who-has 192.0.2.1 tell 192.0.2.2, length 28
The vswitch sees the ARP request transmitted towards the 3012, but no ARP reply is received by the vswitch:
~ # pktcap-uw --uplink vmnic0 --vlan 4035 --stage 0 --dir 0 & pktcap-uw --uplink vmnic0 --vlan 4035 --stage 0 --dir 1
The name of the uplink is vmnic0
The session filter VLAN is 4035
The Stage is Pre
The dir is Rx
No server port specifed, select 13149 as the port
Output the packet info to console.
Local CID 2
Listen on port 13149
Accept...Vsock connection from port 1054 cid 2
The name of the uplink is vmnic0
The session filter VLAN is 4035
The Stage is Pre
The dir is Tx
No server port specifed, select 45918 as the port
Output the packet info to console.
Local CID 2
Listen on port 45918
Accept...Vsock connection from port 1055 cid 2
14:36:18.611941[1] Captured at PortOutput point, TSO not enabled, Checksum not offloaded and not verified, VLAN tag 4035, length 60.
Segment[0] ---- 60 bytes:
0x0000: ffff ffff ffff 000c 2922 09e0 0806 0001
0x0010: 0800 0604 0001 000c 2922 09e0 c000 0202
0x0020: 0000 0000 0000 c000 0201 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
14:36:19.609276[2] Captured at PortOutput point, TSO not enabled, Checksum not offloaded and not verified, VLAN tag 4035, length 60.
Segment[0] ---- 60 bytes:
0x0000: ffff ffff ffff 000c 2922 09e0 0806 0001
0x0010: 0800 0604 0001 000c 2922 09e0 c000 0202
0x0020: 0000 0000 0000 c000 0201 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
14:36:20.609278[3] Captured at PortOutput point, TSO not enabled, Checksum not offloaded and not verified, VLAN tag 4035, length 60.
Segment[0] ---- 60 bytes:
0x0000: ffff ffff ffff 000c 2922 09e0 0806 0001
0x0010: 0800 0604 0001 000c 2922 09e0 c000 0202
0x0020: 0000 0000 0000 c000 0201 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
The 3012 see the ARP request received, and generates an ARP reply:
ics01.ht-tor-mow-01#
Aug 20 10:39:57.895: IP ARP: rcvd req src 192.0.2.2 000c.2922.09e0, dst 192.0.2.1 Vlan4035
Aug 20 10:39:57.895: IP ARP: sent rep src 192.0.2.1 d057.4c0e.92c2,
dst 192.0.2.2 000c.2922.09e0 Vlan4035
Aug 20 10:39:58.894: IP ARP: rcvd req src 192.0.2.2 000c.2922.09e0, dst 192.0.2.1 Vlan4035
Aug 20 10:39:58.894: IP ARP: sent rep src 192.0.2.1 d057.4c0e.92c2,
dst 192.0.2.2 000c.2922.09e0 Vlan4035
Aug 20 10:39:59.892: IP ARP: rcvd req src 192.0.2.2 000c.2922.09e0, dst 192.0.2.1 Vlan4035
Aug 20 10:39:59.892: IP ARP: sent rep src 192.0.2.1 d057.4c0e.92c2,
dst 192.0.2.2 000c.2922.09e0 Vlan4035
ARP entry on VM for blade switch SVI is incomplete:
root@monitor:/home/jlixfeld# arp -an
? (192.0.2.1) at <incomplete> on eth1
ARP entry on blade switch for VM is complete:
ics01.ht-tor-mow-01#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.0.2.2 0 000c.2922.09e0 ARPA Vlan4035
Internet 192.0.2.1 - d057.4c0e.92c2 ARPA Vlan4035
For completeness, I tried to create a static ARP entry on the VM for the MAC of the the Vlan4035 SVI on the 3012, but traffic from the 3012 to the VM is still not seen by the vswitch. That is, if I ping from the VM to the SVI, the vswitch sees the ECHO REQUEST transmitted towards the 3012, but doesn't see the ECHO REPLY come back. Conversely, the vswitch does not see an ECHO REQUEST generated from the 3012 towards the VM.
So, to me it looks like there's a unidirectional traffic condition happening between the 3012 and the VM. Traffic from the VM to the 3012 seems fine, but not in the opposite direction (except STP. STP traffic is seen by the vswitch).
I have tried this on 2 other VMs on the same HS22. They can all successfully reach each other via the switch, but none of them can reach the SVI on the 3012.
I have tried different VLANs and different subnets with no success.
This was working up until a few days ago, which I changed some stuff around on one of the VMs. It worked for a while, then stopped, and I haven't been able to get it going again. I've rebooted VMs, deleted and re-added the affected VLANs, bounced Gi0/3 facing the blade, defaulted the configuration of Gi0/3.
The settings in vSphere seem to be OK.
Any thoughts or ideas?