Author: Adrian Page 10 of 13

Repair docker application engine in vRealize Automation 8

Recently, services didn’t come back up again on vRA 8.6 single node installation after a power loss in my homelab.

The connection to Kubernetes was refused and failed with the following error while executing kubectl get pods:

The connection to the server vra-k8s.local:6443 was refused - did you specify the right host or port?

I was finally able to fix it using the procedure described in VMware KB article 82378:

  1. Take a snapshot of the vRA VM.
  2. Locate an etcd backup at /data/etcd-backup/ and copy the selected backup to /root
  3. Reset Kubernetes by running vracli cluster leave
  4. Restore the etcd backup in /root by using the /opt/scripts/recover_etcd.sh command.
    Example: /opt/scripts/recover_etcd.sh --confirm /root/backup-123456789.db
  5. Extract VA config from etcd with
    kubectl get vaconfig -o yaml > /root/vaconfig.yaml
  6. Reset Kubernetes once again using
    vracli cluster leave
  7. Run to Install the VA config
    kubectl apply -f /root/vaconfig.yaml --force
  8. Run vracli license to confirm that VA config is installed properly.
  9. Run
    /opt/scripts/deploy.sh

Shutdown of vSphere with Tanzu, vRealize Suite, VCD

We have to shutdown the management components of the VMware homelab in a specific order to keep components operational by maintaining the necessary infrastructure, networking, and management services prior before shutdown.

The order is as follows:

Replace certificates for VMware Cloud Director portal and console proxy

With VMware Cloud Director 10.3, the SSL certificates of the HTTP and the console proxy endpoints are not stored in a PKCS12/JCEKS formatted keystore anymore, but in PEM format only.

So the simplified process is as follows (after creating a backup of the old certificate and key files):

/opt/vmware/vcloud-director/bin/cell-management-tool certificates -j --cert /opt/vmware/vcloud-director/data/transfer/user.http.pem --key /opt/vmware/vcloud-director/data/transfer/user.http.key

/opt/vmware/vcloud-director/bin/cell-management-tool certificates -p --cert /opt/vmware/vcloud-director/data/transfer/user.consoleproxy.pem --key /opt/vmware/vcloud-director/data/transfer/user.consoleproxy.key

Afterwards the cell must be restarted, e.g. using service vmware-vcd restart. The procedure must be performed on all available cells in a cell group.

To customize the VMware Cloud Director URLs we must also edit the Web Portal endpoints in the Service Provider Admin Portal under Administration > Settings > Public Addresses.
There we enter our VMware Cloud Director public URL for HTTPS (secure) connections and click Replace Certificate File to upload the certificates that establish the trust chain for that endpoint. The certificate chain must match the certificate used by the service endpoint, which is the certificate we’ve uploaded to each VMware Cloud Director cell before.

Pro tip: The key must be provided in a separate file, not included in the certificate PEM file. The certificate file must have the following schema:

-----BEGIN CERTIFICATE-----
(Your Primary SSL certificate: DomainName.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Intermediate certificate: CA.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Root certificate: TrustedRoot.crt)
-----END CERTIFICATE-----

VMware homelab vRealize Suite setup

VMware vRealize Suite is a purpose-built management solution for the heterogeneous data center and the hybrid cloud. It delivers and manages infrastructure and applications to increase the business agility while maintaining IT control. It provides the most comprehensive management stack for private and public clouds, multiple hypervisors, and physical infrastructure.

It consists of the following solutions:

  • vRealize Automation (and vRealize Orchestrator)
  • vRealize Operations
  • vRealize Log Insight

To automate installation, configuration, upgrade, patch, configuration management, drift remediation and health from within a single pane of glass, we will use vRealize Suite Lifecycle Manager.

The below diagram shows technological capabilities and organizational constructs.

In the lab environment we’ll install all solutions as single node instances with the following sizings:

NamePurposeSizevCPUMemory (GB)Disk (GB)
vrslcm1Lifecycle Manager2678
wsa1avIDMMedium81660
vra1avRealize AutomationMedium1242236
vrops1avRealize OperationsExtra small28274
vrli1avRealize Log InsightSmall48530

After the deployment of these solutions, we’re going to initially integrate them.

VMware homelab VCD setup

With VMware VMware Cloud Director you can build secure, multi-tenant clouds by pooling virtual infrastructure resources into virtual data centers and exposing them to users through Web-based portals and programmatic interfaces as a fully automated, catalog-based service.

In the lab environment, we’ll setup a simple single cell installation, and add our workload vCenter Server vc2.lab.local and the NSX-T Manager nsx1.lab.local as infrastructure resources.

From these infrastructures we’ll create cloud resources such as a provider VDC, a Geneve network pool, and an External network.

Then we’ll create a tenant organization and assign resources from the provider VDC as an organization VDC to this particular organization. We’ll also create an Edge Gateway to allow the tenant to access the outside world from within his Cloud.

Page 10 of 13

All your base are belong to us.