ESXi 6.7 PSOD with qlfe3 driver version above 1.0.69.1

Had a ESXi PSOD today. That does not happened that often, so I was quite surprised to find out that it was not a hardware related issue that was the root cause.

VMware did an analysis of the memory dump, and it turned out to be a faulty driver. That made sense since the PSOD often comes from drivers og agents when it is not a hardware issue.

The PSOD i got was the following:

#PF Exception 14 in World xxxxxxx:vmnicX-pollw IP xxxxxxxxxx addr xxxxxxxx
Continue reading ESXi 6.7 PSOD with qlfe3 driver version above 1.0.69.1

Error: cannot install the vcenter agent service. cannot upload agent after vCSA upgrade

I was just updating a vCenter server and some ESXi hosts, but after running the vCenter update I found vCenter full of HA Agent install fails. To stop this fail loop, I turned off VMware HA while figuring out what was wrong.

Error: cannot install the vcenter agent service. cannot upload agent after vCSA upgrade
Continue reading Error: cannot install the vcenter agent service. cannot upload agent after vCSA upgrade

Update Manager ELX_bootbank_elx-esx-libelxima.so driver conflict

Just provisioned the HPE ESXi 6.7 Update 3 custom OEM image onto some HP DL560 Gen10 servers.

After I updated the servers using update manager and the HPE vibsdepot I ran into problems. Turns out there is a conflict between the VMware provided driver and the HPE provided driver.

The result is that I cannot install all updates to satisfy compliance.

Checking the esxupdate.log file on the ESXi hosts I get the following error:

ValueError: VIBs ELX_bootbank_elx-esx-libelxima.so_12.0.1108.0-03 and ELX_bootbank_elx-esx-libelxima.so_12.0.1108.0-03 have unequal values of the 'payloads' attribute: '[elx-esx-libelxi: 1602.936 KB]' != '[elx-esx-libelxi: 1493.833 KB]'
Continue reading Update Manager ELX_bootbank_elx-esx-libelxima.so driver conflict

How to upgrade to VMware Tools 10.3.10

So VMware released a security advisory (VMSA-2019-0009) about vulnerable VMware Tools.

In short it is recommended to upgrade all VMware Tools to at least version 10.3.10 (Build 10346) which is the newest at the time of writing.

After patching ESXi to the latest build, you might realize that the VMware tools you are left with, after upgrading the tools, in a VM, is 10.3.5 (build 10341) Continue reading How to upgrade to VMware Tools 10.3.10

vSAN – Downgrading NVMe driver in ESXi 6.7 Update 1

Recently ran into a HPE Proliant m510 server running vSAN, where vSAN complained that the controller driver for the NVMe disk where too new.

The health error said that the current driver nvme (1.2.2.17.-1vmw.670.1.28.10302608) was to new and the recommended driver was nvme (1.2.1.34-1vmw.670.0.08169922)

Downgrading is not always a breeze. When going to VMware compatibility guide, the NVMe disk is supported for vSAN 6.7 Update 1, and there are no download links to a specific driver, so how do you get the old driver? Continue reading vSAN – Downgrading NVMe driver in ESXi 6.7 Update 1

10fb does not support flow control autoneg

Issue with nic driver on HPE servers after updating HPE drivers on ESXi 6.5 and 6.7

What happened

I ran into an issue the other day with a vCenter Server Appliance filling up one of its partitions. The partition that was filling up was the /storage/seat partition. This partition holds the postgres SQL database, so the vCenter server was in trouble.

After some digging around I realized that the root cause was a new event error from all ESXi hosts, that was coming at a rapid pace. The errors had started during the last driver and base updates, and only the HPE servers was affected. Continue reading 10fb does not support flow control autoneg

Host xxx.xxx.xxx.xxx is not compatible with the VDS version

Just had an odd issue today.

A customer had created a Virtual Distributed Switch, but was unable to add his ESXi hosts to the vDS. It said that: “Host is not compatible with the VDS version.”

He was only able to join his version 6.5 ESXi host to a 5.5 vDS. If it was upgraded to version 6.0 or 6.5 it did not work.

There are multiple reports of this online related to upgraded hosts and vCenters. I suspect that it is an issue that you only run into if you do major upgrades without reinstalling ESXi, and since I never do that I have not had that problem before.

The quick solution to this problem is: Continue reading Host xxx.xxx.xxx.xxx is not compatible with the VDS version

Unable to VMotion to new Lenovo SR650 Host

Hi,

I have had an annoying issues at two customer sites now, and I want to share the solution with you.

The problem is that you cannot VMotion VMs to a newly installed ESXi 6.5 hosts running on Lenovo SR650 hardware. The CPU used in the new host is Intel Xeon Gold 6154 Processor, and the old hosts are using Intel Xeon Processor E7-4880 v2. I do not think that the source CPU model is relevant to the issue it could be any supported Intel CPU in the same cpu family.

When trying to VMotion the following error is displayed:

The virtual machine requires hardware features that are unsupported or disabled on the target host:
"""""""""""""* General incompatibilities
"
If possible, use a cluster with Enhanced vMotion Compatibility (EVC) enabled; see KB article 1003212.

CPUID details: incompatibility at level 0x1 register 'ecx'.
Host bits: 0110:0010:1101:1000:0011:0010:0000:0011
Required: x1xx:xx1x:10x1:1xx0:xx1x:xx1x:xxxx:xx11

If you then try to enable EVC in the cluster it complains that the new hosts has an issue, and returns this error:

The host's CPU hardware should support the cluster's current Enhanced vMotion Compatibility mode, but some of the necessary CPU features are missing from the host. Check the host's BIOS configuration to ensure that no necessary features are disabled (such as XD, VT, AES, or PCLMULQDQ for Intel, or NX for AMD). For more information, see KB article 1003212.

Continue reading Unable to VMotion to new Lenovo SR650 Host

Cannot add ESXi host to vCenter

Hi,

Had an annoying error today. Was updating an ESXi image for use with AutoDeploy. When I reinstalled the hosts they would not join vCenter. My workflow removes them from vCenter in the process, but they were unable to rejoin, and I could not add them manually either.

I got two error:

When selected the license in the add host wizard I got this error:

Cannot decode the licensed features on the host before it is added to vCenter Server. You might be unable to assign the selected license, because of unsupported features in use or some features might become unavailable after you assign the license.

 

I pushed through, but when the task reaches 100% it gave another error:

License file download from <servername> to vCenter Server failed due to exception: vmodl.fault.SecurityError.

Well to cut a long story short it turned out to be a time issue. Some of the serveres was not allowed to talk to the NTP servers. and their time had drifted. vCenter was located on one of these serveres, and its time was 5-6 minutes behind the ESXi servers that I was trying to join.

The NTP connection issue was corrected. Time was checked on all servers.

Problem solved.

Hope this helps someone.

Fujitsu Primergy ESXi Install Server Notes – BIOS Settings

So I was installing some Fujitsu Primergy RX2530 M4 servers today, and since I mostly work with HPE and Lenovo servers I had lookup the optimal BIOS settings to run ESXi 6.5.

This is what I came up with. From the default settings I only changed a couple of things that I found important. Continue reading Fujitsu Primergy ESXi Install Server Notes – BIOS Settings