We deploy and wipe ESXi servers on an almost weekly basis, as they are used for short term projects and the machine is wiped and reinstalled in readiness for the next project. Although we have vCenter Server, this is used only for a small set of more permanently installed virtualised systems.
Over the last couple of weeks we have been trying to define a process for the installation and creation of a standard ESXi 4.1 host setup on our HP DL380G6 servers.
Now, because the HP DL380G6 runs like a dog without some specific bundles installed on top of the ESXi 4.1 base installation, the process requires running vihostupdate from a Windows machine with the CLI installed. To remove this step we were trying different methods to image a final installation and just write the image to the server to refresh it. The final technique settled on installing ESXi 4.1 to a USB memory stick, which would be plugged into the internal port inside the DL380s. We found a nice bit of software that would reliably duplicate USB memory sticks, that ensures the copies are bootable.
All was well and good, until someone decided they needed TWO ESXi boxes for a particular project.
Anyone spot the problem yet?
The engineer found that when the two servers were on the network, despite having different IP addresses they simply would stay on the network. They would randomly drop of the network and come back, repeatedly. BUT... at no time were both visible on the network at the same time. We had a client machine with two CMD windows running PING continuously, and the responses would only be received by one or the other, and it would seemingly switch over at random intervals as to which was receiving responses. Needless to say trying to connect vSphere Client to one of the servers was a strange experience at best, and nothing constructive could be achieved even if a connection was made.
We resorted to using Wireshark to try and work out what was happening with the ICMP ping packets. We discovered that we could see the ping packets going to the two different IP addresses, but deeper inspection showed that the destinations for both those ping packets had the same MAC address, despite having different IP addresses. So basically our poor little Netgear switch was getting somewhat confused.
After getting confused about why we had a duplicate MAC address, when the NICs clearly had different ones, it suddenly dawned on me what had happened.
When we created the master USB memory stick by installing ESXi to it, it was duplicated in the minutest detail. Right down to the MAC address that had presumably been randomly generated for the VMKernel Management Port.
The solution wasn't all that complicated, create a new VMKernel port on one of the servers, and delete the original one. This is not so onerous doing with the vSphere client, but I would like to reduce the lists of 'things' that the project engineers need to do when they are given an ESXi host to use.
Is there a way to trigger a VMKernel port to have a newly assigned MAC address, AFTER it has been created? Preferably through some sort of command or script that could be run from the Tech Support Mode console?