Hi,

I have configured syslog logging for my ESXI 6.5 hosts.

I have configured loglevel info for all syslog/vxa related entries in advanced settings.

My syslog server is filled up with crap (approx 100k messages a day from 1 host!)

I know I can edit the RHTTPPROXY config file and change it from verbose to info, but I cant find anything related to the HOSTD (and it should not be necessary to change the xml config files for something like this)...

Has anyone seen this before?

Thanks,

HI!

I was using ESXI 6.5 (and after my problem, was upgraded to 6.7U2) with Intel NVME SSD DC P3520 Series, and when I started a Virtual Machine with Windows Server (with vmware tools and paravirtual scsi controller), in vmkernel.log I received alot of messages like this:

2020-02-13T12:23:18.666Z cpu3:2097600)ScsiDeviceIO: 3449: Cmd(0x459a40d29440) 0x93, CmdSN 0x56f from world 2097305 to dev "t10.NVMe____INTEL_SSDPE2MX450G7_____________________CVPF6334002A450RGN__00000001" failed H:0x0 D:0x2 P:0x0 Valid sense data:

2020-02-13T12:23:18.666Z cpu3:2097600)0x5 0x0 0x0.

2020-02-13T12:23:18.673Z cpu3:2097600)ScsiDeviceIO: 3449: Cmd(0x459a40d2a700) 0x93, CmdSN 0x571 from world 2097305 to dev "t10.NVMe____INTEL_SSDPE2MX450G7_____________________CVPF6334002A450RGN__00000001" failed H:0x0 D:0x2 P:0x0 Valid sense data:

2020-02-13T12:23:18.673Z cpu3:2097600)0x5 0x0 0x0.

2020-02-13T12:23:18.677Z cpu3:2097600)ScsiDeviceIO: 3449: Cmd(0x459a40cbbd00) 0x93, CmdSN 0x573 from world 2097305 to dev "t10.NVMe____INTEL_SSDPE2MX450G7_____________________CVPF6334002A450RGN__00000001" failed H:0x0 D:0x2 P:0x0 Valid sense data:

2020-02-13T12:23:18.677Z cpu3:2097600)0x5 0x0 0x0.

2020-02-13T12:24:03.442Z cpu0:2097643)HPP: HppResetDeviceLogThrottling:631: last error status from device t10.NVMe____INTEL_SSDPE2MX450G7_____________________CVPF6334002A450RGN__00000001 repeated4 times

 

I was upgrade firmware and install intel specifc vib (https://www.intel.com/content/dam/support/us/en/documents/memory-and-storage/ssd-software/Intel_VMD_NVMe_VMWare_User_Gui… ), I was check the Sense codes (https://www.virten.net/vmware/esxi-scsi-sense-code-decoder/?host=0&device=2&plugin=0&sensekey=5&asc=0&ascq=0&opcode=93 ), and I was follow the HPP best pratice (VMware High Performance Plug-In ), but I can´t solve this problem.

And after some time, start this error in same log:

2020-02-13T13:09:09.893Z cpu4:2097183)HPP: HppThrottleLogForDevice:507: Error status H:0xc D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. from device t10.NVMe____INTEL_SSDPE2MX450G7_____________________CVPF6334002A450RGN__00000001 repeated 1280 times
2020-02-13T13:09:09.896Z cpu4:2097690)0000:03:00.0 nvme_ScsiCommand Failed Dsm request
2020-02-13T13:09:09.896Z cpu4:2097690)0000:03:00.0 nvme_ScsiCommand Failed Dsm request
2020-02-13T13:09:09.899Z cpu4:2097690)0000:03:00.0 nvme_ScsiCommand Failed Dsm request
2020-02-13T13:09:09.899Z cpu4:2097690)0000:03:00.0 nvme_ScsiCommand Failed Dsm request
2020-02-13T13:09:09.902Z cpu4:2097690)0000:03:00.0 nvme_ScsiCommand Failed Dsm request
2020-02-13T13:09:09.902Z cpu4:2097690)0000:03:00.0 nvme_ScsiCommand Failed Dsm request
2020-02-13T13:09:09.907Z cpu4:2097690)0000:03:00.0 nvme_ScsiCommand Failed Dsm request
2020-02-13T13:09:09.907Z cpu4:2097690)0000:03:00.0 nvme_ScsiCommand Failed Dsm request

and, after start this error, I stop all vmware, put in maintence mode and this error don´t stop. I need restart esxi to stop.

can Somebody help?

Tks for everthing!

Hi all,

I have ISPconfig with few personal websites and I got Comodo Positive SSL Wildcard. Using ISPconfigI went thru process generating CSR for *.mydomain.com and sending it to where I got SSL from the re seller, I got it back and pasted into ISPconfig, so far so good.

When I do the same for ESXi which is on different IP esxi.mydomain.com I have to use this guide and OpenSSL for windows https://www.comprofix.com/2017/03/02/using-letsencrypt-esxi-vps/

Since when I run ESXi command I get:

[root@esxi:/vmfs/volumes/59c20232-9fad620f-8e7c-0cc47a0c8c1c/verticalbackup] openssl req -x509 -sha256 -newkey rsa:2048 -keyout rui.key -config

openssl.cfg -out rui.crt -days 3650

error on line -1 of openssl.cfg

1022206424744:error:02001002:system library:fopen:No such file or directory:bss_file.c:175:fopen('openssl.cfg','rb')

1022206424744:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:182:

1022206424744:error:0E078072:configuration file routines:DEF_LOAD:no such file:conf_def.c:195:

So when I upload CRT to SSL re seller and get it back from Comodo I get STAR_mydomain_com.crt.crt file and I replace rui.crt in /etc/vmware/ssl and do services.sh restart

I no longer can access https://esxi.mydomain.com and I have to revert to Let's Encrypt certs to log back in.

Any idea?

Hello,

I have the following hardware:

Mother Board: Advantech ASMB-925-00A1

CPU: Gold Processor: 5218N  x2

Memory: 16Gb x4

Raid: 96RC-SAS-8P

HDD: Toshiba NLS 2T  x6

GPU: NVIDIA Quadro TRX4000

After installation of vSphere 6.7 U3 when reboot or shutdown the computer on the first boot I have a pink screen and after I power off the
computer and turn On again the vSphere start to work.

The error:

Need your help,

Thanks,

Moshe.
   
 
 
 
 
 
 
 
 
 
 
 

Problem scope and background

ESXi 6.x after last unclear shuttdown stopped to boot. Loading was hanging on "ntfs4client" screen.

Alt+F1 showed following error message:

jumpstart: bora/lib/vmkctl/system/SystemInfoImpl.cpp:1511: virtual void VmkCtl::System::SystemInfoImpl::SetupSymlinks(bool, bool): Assertion `locker.get()` failed.

Reason

Problem leads to damaged swap file, most likely because of power-outage, swap file is marked as 'locked' and hypervisor couldn't deal with situation. Other possible reason is due to Swap Parser bug - due to file corruption, we see Assertion Failure and process silently quits.

Solution

Boot ESXi installer. Hit ALT+F1 to open Console.

Enter "root" and press Enter twice.

cd to /vmfs/volume/

cd to system datastore (which contains swp files/vswp files)

rm *.swp

rm *.vswp

Not sure if need to remove just a *.swp or both *.swp *.vswp - i removed both and booting went OK, having no excess time checking the exact nature of problem.

Best Regards,

Piotr Karwowski

Hi,

ESX 6.7 U2 with the latest update. This is a server machine, 2 Xeon, it has one Nvidia 2080 card. At one point I had a VM using the GPU, all was good. But I had to reinstalled ESX from scratch ( change to bigger disk), and now I can't get it to work again.    Note that I have 2 other computers (different model) with V100 cards that are working ok so I'm assuming I know what needs to be done to get this working:

hypervisor.cpuid.v0 = "FALSE" 

pciPassthru.use64bitMMIO = "TRUE"  

pciPassthru.64bitMMIOSizeGB = "32"

However the VM won't start. Noticed the "Failed to adjust IOMMU: Failure" "PANIC: PCIPassthru: failed to adjust IOMMU mappings"

020-02-13T23:59:52.927Z| vcpu-0| I125: Transitioned vmx/execState/val to poweredOn

2020-02-13T23:59:52.927Z| vcpu-0| I125: SCSIFilterSBDAttachCBRC: device scsi0:0 is not SBD. Skipping CBRC attach SBD way.

2020-02-13T23:59:52.927Z| vcpu-0| I125: Tools: Adding Tools inactivity timer.

2020-02-13T23:59:52.927Z| vcpu-0| I125: Intel VT: FlexPriority enabled.

2020-02-13T23:59:52.927Z| vcpu-0| I125: Intel VT: VPID enabled.

2020-02-13T23:59:52.929Z| vcpu-0| I125: Intel VT enabled.

2020-02-13T23:59:52.950Z| vcpu-0| I125: Failed to adjust IOMMU: Failure

2020-02-13T23:59:52.950Z| vcpu-0| E105: PANIC: PCIPassthru: failed to adjust IOMMU mappings.

2020-02-13T23:59:53.922Z| vcpu-0| W115: A core file is available in "/vmfs/volumes/5d606047-b33c61be-dfd6-ac1f6babb1ac/Malcom2080/vmx-zdump.001"

2020-02-13T23:59:53.925Z| vcpu-0| I125: Writing monitor file `vmmcores.gz`

2020-02-13T23:59:53.926Z| mks| W115: Panic in progress... ungrabbing

2020-02-13T23:59:53.926Z| mks| I125: MKS: Release starting (Panic)

2020-02-13T23:59:53.926Z| mks| I125: MKS: Release finished (Panic)

2020-02-13T23:59:53.928Z| vcpu-0| W115: Dumping core for vcpu-0

2020-02-13T23:59:53.928Z| vcpu-0| I125: VMK Stack for vcpu 0 is at 0x451a1e993000

2020-02-13T23:59:53.928Z| vcpu-0| I125: Beginning monitor coredump

If I remove the gpu from the virtual machine it starts just fine.

Suggestions?

Mario

Hi dear Vmware community!  I'm a rookie with ESXI and faced with a problem. I have an issue with my ESXI work which i will describe below and hope someone could help me with this, thank you all in advance!

So my setup:

- CPU Intel Xeon Silver 4214

- Supermicro MBD-X11DPL-I-O - ATX

- Supermicro NMVe AOC-SLG3-2M2 with two Samsung M2 disks in it (Samsung 970 Evo plus 1TB)

- 4 pieces of Crucial 32GB DDR4-2666 RDIMM

Esxi runs from USB flash drive.

I installed Intel-nvme-vmd (1.8.0.1001-1oem.670.0.0.8169922) driver for vmhba adapter (with the driver by default iavmd_1.2.0.1011-2vmw.670.0.0.8169922 it fails to BSOD )

The problem i will describe below occurs after some period of time (as usual after ~20 hours of working under usual load) i turned HOST on and launch set of my Virtual machines.

All the VMs are configured identically and stored on different datastores  (some Vms on 1st, some on 2nd disk) :

Disk - Thick provisioned, eagerly zeroed with Nvme Controller

Network - VMxnet 3

Others option mostly default..

Free space on both storage is more than 50%

So the problem is - Very high latency on monitor:

i started dig dipper and found very high Kavg during see high latency above (It easily jumps to 1000+)

What is strange - this high KAVG appears only on certain SSD (let it be  marked with "A") disk. I tried change places for disks in PCI card, and high KAVG was still on those disk A.

At the same time i do not see any Queue take place:

Also it seems to me that latency on Vms is pretty normal...

I switched off hardware acceleration (Vaai), but it did not help.

I attached 2 log files:

- vmkernelv

- mkwarning

There are a lot of errors and warnings like:

HppThrottleLogForDevice:564: Cmd 0x42 (0x459a4233da40, 2108176) to dev "t10.NVMe____Samsung_SSD_970_EVO_Plus_1TB____________S4EWNF0M717097A_____00000001" on path "vmhba2:C0:T1:L0" Failed:

2020-02-14T08:56:48.253Z cpu3:2097198)WARNING: HPP: HppThrottleLogForDevice:570: Error status H:0xc D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.

In case any other logs required i will provide for analyzing the issue. I will really-really appreciate any help!

Hello everyone,

I just added a new Host (same product) on my cluster and i want to know if have a tool can export esxi configuration to import on my new esxi host ?

Thanks for help.

I had setup a 10TB Western Digital USB drive as a datastore and that worked fine.  I just updated from 6.5 to 6.5 Update 3 and now it's seen but not working.

I installed a 2nd nic, setup the port group, switch, ip address and set it up as management and can manage ESXi from it.  When I create a vm I don't get the option to use it.

Not sure what went wrong, but all of a sudden, I can't use my keyboard to control any VMs via their web console.

Using the latest chrome normal keys (a-z,0-9) do not generate any response from the vm, whereas up and down keys scroll up and down on the console's history, even if the VM is showing a setup screen which normally would intercept these inputs.

While using non chromium MS edge, keyboard inputs work as they should.

Looking at the developer console (both edge & chrome), I see many errors for "Unknown class found TooManyWrites" that is triggered from main.js

Any idea what could have gone wrong? This setup was working before.

Hi Guys,

I've recently installed a Nvidia Tesla T4 GPU in my server running ESXi 6.5.

I've tried making a passthrough on all 32 addresses for the card on the list but when that happens and when and when I try to assign the card to a VM, the option the assign the PCI device is greyed out.

I'e managed to to set a passthrough and make Active 1 address and than I was managed to assign this card to the VM.

My question is:

1. Does this mean that only 1 GPU core is assigned to the VM out of a possible 32?

2. If so, any ideas how I'm able to assign all 32 cores of this card to the VM as when I try the option to add the PCI device to the VM is greyed out?

Please advise.

Kind Regards

GMSS

I have recently acquired a new Ryzen desktop based on the X370 chipset and its associated SATA controller. I was having problems in high I/O scenarios, so I tried to use an old SAS controller I had (Dell SAS 6/ir.) I attached my SATA hard drive to the SAS controller and it booted off the SATA hard drive fine. However, after booting the datastore was not available. The SAS HBA and the hard drive was visible, and I could view the partition layout of the drive. But nothing I did would allow me to access the old datastore. I eventually reconnected to the motherboard's onboard SATA controller, and my old VMs and datastore were once again available.

The fact that I can boot indicates that ESXi can access the hard drive, but something is preventing it from seeing the datastore after booting. Or maybe I'm missing a step in adding the datastore. Any suggestions?

Hi,

Having build a new lab server with:

Gigabyte C246M-WU4 Motherboard

128GB Corsair RAM

Intel Xeon E-2176G CPU

Samsung 970 PRO NVMe SSD

Fresh install of ESXI 6.7.0 Update 3 (free version)

I often experience that the host completely freezes with the black/white screen also frozen. No VM's are running, no ping/login to host. Num Lock doesn't work. F2 doesn't work. Everything has freezed.

I know that only the chipset + CPU are on the compatbility list, but I figured that if ESXI could run off an old HP laptop, then why not this machine.

Any suggestions to, how to find the root cause of the freezes? I'd really hate having to change to virtualbox

Hi,

We are running vSphere 6.0.  We find that for a number of VMs running Windows Server 2012R2.

It is pretty slow to download Windows Update and the update process takes a long time.

Also, when we click "Reboot Server", the Windows Server seems to lose Network Connectivity but it takes a long time before it enters "Reboot" stage.  Not to mention, it takes a long time for preparing Windows Update.

There is sufficient vCPU / RAM and VMDK file space.

Would it be due to the HDD is thin provisioning  (Others with Thin Provisioning are performing good) ?  Is there any clue from VMware point of view ?

Thanks

Hello,

I've got an odd problem with a recently bought HPE DL360p Gen8 server. VMware seem to have problems reading out serveral hardware sensors like fans and powersupply. The show up as "unspecified". The iLO can read the sensors without any problems.

I've installed the latest offical custom image from HP (VMware-ESXi-6.5.0-Update3-14990892-HPE-preGen9-650.U3.9.6.10.1-Dec2019). I also tried installing older versions like U1 & U2 but the results are the same. I'm running out of ideas and I can't find anything related to this error in the internet.

iLO is updated to 2.73 (Feb 11 2020)

System ROM version: P71 05/24/2019

Hostd.log shows nothing unusual regarding IPMI:

IpmiIfcRhGetDeviceId: BMC mfg is Hewlett-Packard FW: 2.73 count_events: starting communication with bmc over ipmi driver count_events: GET_SEL_REPO_INFO returned {version: 0x51, count 64, free 0,add_stamp 1581853816, erase_stamp 1581853816 op_support 2} IPMI SEL sync took 0 seconds 0 sel records, last 164

What could cause this problem? Where and what can I check?

Any help is appreciated. Thank you very much in advance!

My apologies, I am new to vSphere.  When trying to create the first datastore on a new vsphere 6.7 installation.  Under storage section (left menu), then on Datastore tab, it does not show any local disk storage  to make a new datastore. Under the adapters it lists the H700 with unknown status. Under devices tab, it does show both logical storage units from the H700, one is 2.18TB capacity and another is is 300gb, which is the total of my internal local storage.  If esxi cannot read the H700 status, then maybe it also cannot see drives with available storage.

When the server is booted to Win10 it can see all storage so I believe the storage configuration is functioning.

Is it possible to work around this be installing ver 6.5 of esxi?  Or is a different raid adapter required?  Other recommendations are appreciated.

I've recently moved our main backup system from Backup Exec 2014 to Backup Exec 15.  It has gone horribly.  (Not my idea to use Backup Exec.  I was very familiar with it, and hadn't had too many issues with it previously, but I still knew its reputation.  This decision was made before I joined the company.  Not wanting to make waves, I went along with it and tuned up the old 2014 setup, and made the plans to move to the new BE 15 setup.)

On BE 2014, I managed to adjust things to get the backups stable and performing OK.  They weren't great, but they were OK.  The old backup server only had 1Gb networking, and the previous admin had it set up with 2 1Gb links software bonded (Windows 2012r2 link bonding) and the iSCSI storage also used just a 1Gb link.  It was able to back up even our largest file shares in just over 28 hours, so all the full backups could get done over a weekend.  Mostly, backups would average 500-600MB/min. 

I start out with the new backup system, the exact same server and storage, but with 10Gb networking using an Intel dual port X520 and Cisco Nexus switches, one link for network and the other for iSCSI traffic.  The first 4 backups I put on there worked perfectly, and I was seeing 1.3-1.8 Gb traffic on the server network side and 400-700Mb/s traffic on the iSCSI side.  The actual backups, 3 VMs and one physical machine, are showing 1300-1700MB/min speeds in Backup Exec.  No problem, I figure.  This should make things a lot better.  It's tested and ready to take on the rest of the backups.

Not so much.

Upon moving the rest of the servers (VMs and physical) over to the new backup system, it slowed down horribly.  The physical machines backup slowly, at about 600-700MB/min, but they aren't horrible.  For the VMs, though, I'm getting 180-240MB/min for all the backups, including our big file servers.  It is taking upwards of 5 days to complete the jobs.  Even the previously fast backups have slowed to the same crawl.  I've tried both direct VM backups without GRT, VM backups with GRT, and agent backups from the VM OS.  All go at the exact same crawl.  The VM hosts aren't even close to taxed.  I can run 10 VM backups simultaneously and not impact the performance of the VMs' services at all.  In fact, there's more than enough performance to spare.  The VM OSes (Windows 2008r2) report less than 20% (dual) CPU usage, less than 10% disk busy time, and less than 10% (~100Mb) network usage.  The backup storage isn't getting taxed, either.  The backup storage shows less than 1Mb of activity over the 10Gb link.  The storage shows less than near zero busy time on the RAID. 

I've gone to Veritas's support, and they have nothing.  They were the ones who suggested switching up to non-GRT VM backups, and later direct agent backups through the VM's OS.  None of their suggestions have helped at all.  They decided to write off the support ticket saying we have inferior backup storage, but the storage sees almost no activity.  They've completely abandoned me.  I'm about a week away from having to roll back, and my job is likely in danger because of it.  (It's not totally this issue threatening my job.  Long story, but I am good at many things, just not the stuff we do at this company.  I'm certainly not dumb.)

I'm at my wit's end.  I can't figure this out, and Veritas can't figure it out.  Has anyone else out there seen this behavior from BE 15?  Is there a solution to this? 

Hi experts,

       In dual SAN fabric of vmware,  If replace one SAN switch in one SAN fabric,  We only create same zone configuration then the ESXi will automatically restore the  replaced path?  Is it relative to  HBA parameters ? many Thanks.