Category: Tips and Tricks

vCenter Bind Request Failed Error 49 persists after Password Reset?

Sometimes you run into a vCenter issue where the situation is not just broken, but dangerously broken.

This is one of those cases.

If your vCenter is throwing vmdird authentication failures like the ones below, and the normal machine account password reset procedure does not fix it, your vCenter may already be in a very bad state:

err vmdird t@140245530842880: Bind Request Failed (x.x.x.x) error 49: Protocol version: 3, Bind DN: "cn=vcsa,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL
err vmdird t@140245530842880: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)

Let me be very clear: this is a serious recovery situation.

Any remediation from this point is potentially destructive and provided as-is with no guarantee of success. You should assume that services may fail to return cleanly, additional repair steps may be required, certificate repair or endpoint re-registration may become necessary, and in the worst case full recovery may fail.

So do not treat this as a casual “run a command and move on” type of issue.

How To Easily Master vSphere Desired-State Cluster Configuration Files

Introduction

VMware By Broadcom has moved away from the old, clunky Host Profiles in favor of the modern Desired State Cluster Configuration. This “Desired-State” model is fantastic for consistency, but it introduces a new challenge: managing the massive JSON documents that define your cluster’s state.

When you need to scale a cluster, you’re often stuck manually editing host-specific overrides for IPs and hostnames inside a complex JSON structure. I built ClusterConfigForge to turn that manual grind into a streamlined, automated workflow.

The Challenge: Scaling Desired-State Configurations

In the Desired-State model, the entire configuration for a cluster is managed as a single document. While this is great for avoiding “configuration drift,” updating unique host details (like Management/vMotion IPs) for a 32-node cluster still requires tedious, repetitive data entry.

If you’re a consultant or a lead admin, you don’t want to spend your afternoon copy-pasting IPv4 addresses into a text editor. You want a tool that understands the structure and does the heavy lifting for you.

Diagnose Network Problems In A Container Using The Filesystem

So you need to figure out why you application is not working, but your toolset is very limited because you are using either a minimal installation of Linux or you are in a container. This technique should work either way.

You can use the /dev filesystem to check network connections.

Example 1:
You would like to know if you can reach the host 172.16.0.10 on port 443.

ip=172.16.0.10
port=443
if $((echo > /dev/tcp/$ip/$port) &>/dev/null)
then echo "TCP port $port is open"
else echo "TCP port $port is closed"
fi

The above command will you the answer.

You can also write this as a one-liner

ip=172.16.0.10;port=443; (echo > /dev/tcp/$ip/$port) &>/dev/null && echo "TCP port $port is open" || echo "TCP port $port is closed"

Example 2:

You would like to know what connections are being made to and from the container or host, but you do not have netstat.

Cannot export ISO from vLCM cluster image

When you try to export an ISO file in VMware vCenter from a cluster using single cluster image with vLCM. You will get the following error:

A general system error occurred: Error occurred while exporting ESXi image and/or image document.

The error is accompanied with an error in the vmware-vum-server-#.log file in /var/log/vmware/vmware-updatemgr/vum-server catalog like the following:

2023-06-14T12:21:23.882Z error vmware-vum-server[09453] [Originator@6876 sub=VumVapi::Lib::Utils] [ExportTask 92] Failed to export cluster image from depot. errorCode: 99

In my case I was able to export it as a zip bundle and the corresponding json configuration file exported successfully as well.

The problem lies with vendor signatures, and vmware does not currently have a solution for this unfortunately except that it normally helps to remove the vendor packages attached to the cluster.

https://kb.vmware.com/s/article/91237

More information is available here: https://communities.vmware.com/t5/vCenter-Server-Discussions/Cannot-export-vLCM-image-if-you-use-a-custom-SSL-cert-Non/td-p/2881200/page/2

List VMs with Secure Boot enabled on Windows Server 2022

Since Microsoft released: KB5022842 a lot of customers has experienced Windows Server 2022 not being able to boot. On vSphere 7 this might be a problem if you have installed the patch at enabled secure boot for the server.

More information is available here: VMware KB90947

If you need to find VM that are running Windows Server 2022 and have enabled Secure Boot it is not that easy.

The problem is that your cannot always be sure that the OS selected for the VM is the OS actually installed in the VM. If for instance you installed Windows Server 2022 before is was officially supported in vSphere you might have chosen Windows Server 2019. So you will need to use the OS name that VMware tools are reporting.

But what is VMware tools is not running. That’s a problem.

The following script will find VMs with Secure Boot enabled that are running Windows Server 2022, but also VM’s where we are not certain because VMware Tools is not running.

NSX-T Troubleshooting IDFW rules

So you have migrated to NSX-T 3.2 and you are using IDFW rules to allow users to dynamically gain access when they log in to any physical device in the domain.

Only trouble is that now it is not really working, and VMware did not yet implement a way in the gui to see the effective members of Groups that contains Active Directory members.

Well there is a way you can see who is in the group at least, but there are a couple of steps.

How to find the effective group members

Step one is to identify the rule you are troubleshooting. Make a note of the rule id.

Next find the host the destination VM is running on. You can do this manually in vCenter or use powershell. That’s up to you.

Automating VMware Workstation LAB

I am often working with quite large test environments. Powering on ESXi hosts with nested VMs can be a pain when you need to get it running quickly.

Here are some of my tricks to automating VMware Workstation

Nested or Native

Should you buy dedicated hardware or a OP workstation for you next testing environment. If you are not sharing it with others, this might be useful for you.

History

For many years now VMware Workstation has been my secret weapon an daily tool for just about everything in regards to customer remote connections, test environments as so on.

Recently I needed to do some advanced testing with NSX-V and NSX-T. This required a lot more power than what I normally use so I needed to upgrade my testing platform.

The consideration is always with these things. How much are you going to invest, and what are the benefits. For a long time I have been considering buying 4 Intel NUC PC’s for doing these tests, but the problem is that to get a real setup that is flexible you need to invest a lot. Also it is not very flexible as you need to maintain them, and reinstall them everytime you need to play with a newer or older version.

vCenter services not starting after 6.7 Update 3f upgrade

After upgrading a 3 vCenter Enhanced Link Mode environment a customer experienced the following error:

Server is at a higher functional level (1) than partner (<partner vCenter server>)(0) and cannot perform at a lower level.

When checking the domain functional level of each of the three servers, they all state that they are at level 1. The other server are starting like normal.

One server is not starting the vmdir service and it is also that service that is reporting the error. Most of the other services on a vCenter is dependant on this the vmdir.

I do not have a solution as of now. I might have a reason for the issue, and I might have a workaround. VMware is currently trying to figure out how to fix this.

ESXi 6.7 PSOD with qfle3 driver version above 1.0.69.1

Had a ESXi PSOD today. That does not happened that often, so I was quite surprised to find out that it was not a hardware related issue that was the root cause.

VMware did an analysis of the memory dump, and it turned out to be a faulty driver. That made sense since the PSOD often comes from drivers og agents when it is not a hardware issue.

The PSOD i got was the following:

#PF Exception 14 in World xxxxxxx:vmnicX-pollw IP xxxxxxxxxx addr xxxxxxxx