Unable to VMotion to new Lenovo SR650 Host

Hi,

I have had an annoying issues at two customer sites now, and I want to share the solution with you.

The problem is that you cannot VMotion VMs to a newly installed ESXi 6.5 hosts running on Lenovo SR650 hardware. The CPU used in the new host is Intel Xeon Gold 6154 Processor, and the old hosts are using Intel Xeon Processor E7-4880 v2. I do not think that the source CPU model is relevant to the issue it could be any supported Intel CPU in the same cpu family.

When trying to VMotion the following error is displayed:

The virtual machine requires hardware features that are unsupported or disabled on the target host:
"""""""""""""* General incompatibilities
"
If possible, use a cluster with Enhanced vMotion Compatibility (EVC) enabled; see KB article 1003212.

CPUID details: incompatibility at level 0x1 register 'ecx'.
Host bits: 0110:0010:1101:1000:0011:0010:0000:0011
Required: x1xx:xx1x:10x1:1xx0:xx1x:xx1x:xxxx:xx11

If you then try to enable EVC in the cluster it complains that the new hosts has an issue, and returns this error:

The host's CPU hardware should support the cluster's current Enhanced vMotion Compatibility mode, but some of the necessary CPU features are missing from the host. Check the host's BIOS configuration to ensure that no necessary features are disabled (such as XD, VT, AES, or PCLMULQDQ for Intel, or NX for AMD). For more information, see KB article 1003212.

Solution

The solution is to enable MONITOR/MWAIT in the Lenovo BIOS. This is not enabled by default in the new UEFI BIOS.

After doing this you are able to enable EVC.

Extra – Avoid poor performance

If you are reading this article, you are most likely setting up new Lenovo servers. It is important to note that the default BIOS settings on a Lenovo SR650 server will most likely give you horrible performance from time to time. I had customers experience that VMs would start to become drowsy. But whenever the did a VMotion on the VMs to another host, the problem would go away. This is most likely due to wrong BIOS settings. You should check out this article from Lenovo that explains the correct BIOS settings: https://support.lenovo.com/my/en/solutions/ht115952, and here is the long version with the pretty pictures: https://www.ibm.com/support/home/docdisplay?lndocid=migr-5098217

Relevant vmware kb’s:

https://kb.vmware.com/s/article/1003212 – EVC

https://kb.vmware.com/s/article/1034926 – EVC Compatibility Issue

https://kb.vmware.com/s/article/1029785 – VMotion fails

Cowboy  Trick

If this does not help you can maybe try another trick to get around this issue. But before I tell you what it is, you need to know that it is unsupported, and can result in some VMs crashing, and I will explain why later. So needless to say you are doing it at your own risk, but if you really want to VMotion machines without having to shut them off, which of course is a better and supported solution, you can proceed.

If you still cannot enable EVC, and the CPU’s are compatible according to the EVC kb, and you checked this BIOS on the new hosts 1000 times, and you still have this issue, your can do the following.

1. Add the new host to the cluster with EVC disabled.
2. Shut down the new host (Disconnecting the host does not always work)
3. Enabled EVC. (vCenter cannot check the CPU features of the new host, so it will proceed)
4. Power on the new host, and when it connects your are most likely able to VMotion VMs to it from the old hosts

Why this might crash you VM’s:

The reason this might crash you VM’s it that when the VM’s boot it will detects the features of the CPU, and if the feature that you are missing on the new CPU is in use by the OS or by an application that OS og application will most likely crash. Depending on what CPU feature you are missing, this might be more or less likely.

Best of luck