Recently, I wanted to create a new provider VDC with a vSphere cluster as resource pool in a customer VMware Cloud Director environment. It failed with the error message “[ … ] class com.vmware.vcloud.api.presentation.service BadRequestException”.

While checking the cluster in the resource vCenter Server (version 7.x), the following error was shown in the vSphere Client:

“vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs. vSphere Cluster Service VMs are required to maintain the health of vSphere DRS”

The customer has enabled DRS on the cluster, but the vSphere Cluster Service (vCLS) virtual machines are not getting deployed.

While checking the log files on the VCSA, we came across errors in the ESX Agent Manager logs at /var/log/vmware/eam/eam.log, such as:

“Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.
…
at com.sun.proxy.$Proxy48.loginExtensionByCertificate(Unknown Source)
…“

Especially the line noting the stack trace source as the dynamic proxy class $Proxy48 and its method loginExtensionByCertificate drew my attention. Looks like some certificate issues. After discussing what have been changed recently in the customer’s environment, it turns out that the they have replaced the vCenter Server certificates.

The described issue can happen if there is a mismatch between vpxd-extension certificate stored in VECS and the certificate information stored in vCenter Server Database for EAM extension. To resolve the issue, we must update the extension’s certificate with vCenter Server.

Luckily, a working solution for this problem has been already described in VMware KB article 2112577:

  1. Log into the VCSA using SSH.
  2. Enter the shell.
  3. Run this command to retrieve the vpxd-extension solution user certificate and key:
    mkdir /certificate

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output

    /certificate/vpxd-extension.crt
    /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
  4. Run this command to update the extension’s certificate with vCenter Server.
    python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s -u Administrator@vsphere.local
  5. Restart the VMware ESX Manager service with these commands:
    service-control --stop vmware-eam
    service-control --start vmware-eam

Also pay close attention to the related information in the KB article, as we directly hit the described error “certificate verify failed: Hostname mismatch, certificate is not valid for ‘sdkTunnel'”.

In our case, this error could be ignored because we were getting the error after the message “Successfully updated certificate for “com.vmware.vim.eam” extension” as this message confirms that Extension certificate updated successfully with vCenter Server.