Author: Adrian Page 9 of 12

General vRA troubleshooting

The vRealize Automation appliance is installed on Photon OS since verion 8.0. It includes native Kubernetes to host containerized services.The vRA services run as Docker containers in Kubernetes pods. Each Kubernetes pod hosts one or more containers. Examples of vRA containerized services are:

  • RabbitMQ is the industry-standard message bus used in vRealize Automation.
  • The vRA database is a PostgreSQL database which runs as a pod and uses a Persistent Volume (PV) to store data.
  • The vRealize Orchestrator service runs as a pod hosting two key containers, the control center (manages operations of vRO and plugins) and the vco-server (the orchestration engine).

Another basic container concept is a Namespace. Namespaces are a way to divide Kubernetes cluster resources between multiple users.

As you can see, vRA has many components. Of course, they can fail and some knowledge is required to troubleshoot these scenarios.

Repair docker application engine in vRealize Automation 8

Recently, services didn’t come back up again on vRA 8.6 single node installation after a power loss in my homelab.

The connection to Kubernetes was refused and failed with the following error while executing kubectl get pods:

The connection to the server vra-k8s.local:6443 was refused - did you specify the right host or port?

I was finally able to fix it using the procedure described in VMware KB article 82378:

  1. Take a snapshot of the vRA VM.
  2. Locate an etcd backup at /data/etcd-backup/ and copy the selected backup to /root
  3. Reset Kubernetes by running vracli cluster leave
  4. Restore the etcd backup in /root by using the /opt/scripts/recover_etcd.sh command.
    Example: /opt/scripts/recover_etcd.sh --confirm /root/backup-123456789.db
  5. Extract VA config from etcd with
    kubectl get vaconfig -o yaml > /root/vaconfig.yaml
  6. Reset Kubernetes once again using
    vracli cluster leave
  7. Run to Install the VA config
    kubectl apply -f /root/vaconfig.yaml --force
  8. Run vracli license to confirm that VA config is installed properly.
  9. Run
    /opt/scripts/deploy.sh

Shutdown of vSphere with Tanzu, vRealize Suite, VCD

We have to shutdown the management components of the VMware homelab in a specific order to keep components operational by maintaining the necessary infrastructure, networking, and management services prior before shutdown.

The order is as follows:

Replace certificates for VMware Cloud Director portal and console proxy

With VMware Cloud Director 10.3, the SSL certificates of the HTTP and the console proxy endpoints are not stored in a PKCS12/JCEKS formatted keystore anymore, but in PEM format only.

So the simplified process is as follows (after creating a backup of the old certificate and key files):

/opt/vmware/vcloud-director/bin/cell-management-tool certificates -j --cert /opt/vmware/vcloud-director/data/transfer/user.http.pem --key /opt/vmware/vcloud-director/data/transfer/user.http.key

/opt/vmware/vcloud-director/bin/cell-management-tool certificates -p --cert /opt/vmware/vcloud-director/data/transfer/user.consoleproxy.pem --key /opt/vmware/vcloud-director/data/transfer/user.consoleproxy.key

Afterwards the cell must be restarted, e.g. using service vmware-vcd restart. The procedure must be performed on all available cells in a cell group.

To customize the VMware Cloud Director URLs we must also edit the Web Portal endpoints in the Service Provider Admin Portal under Administration > Settings > Public Addresses.
There we enter our VMware Cloud Director public URL for HTTPS (secure) connections and click Replace Certificate File to upload the certificates that establish the trust chain for that endpoint. The certificate chain must match the certificate used by the service endpoint, which is the certificate we’ve uploaded to each VMware Cloud Director cell before.

Pro tip: The key must be provided in a separate file, not included in the certificate PEM file. The certificate file must have the following schema:

-----BEGIN CERTIFICATE-----
(Your Primary SSL certificate: DomainName.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Intermediate certificate: CA.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Root certificate: TrustedRoot.crt)
-----END CERTIFICATE-----

VMware homelab vRealize Suite setup

VMware vRealize Suite is a purpose-built management solution for the heterogeneous data center and the hybrid cloud. It delivers and manages infrastructure and applications to increase the business agility while maintaining IT control. It provides the most comprehensive management stack for private and public clouds, multiple hypervisors, and physical infrastructure.

It consists of the following solutions:

  • vRealize Automation (and vRealize Orchestrator)
  • vRealize Operations
  • vRealize Log Insight

To automate installation, configuration, upgrade, patch, configuration management, drift remediation and health from within a single pane of glass, we will use vRealize Suite Lifecycle Manager.

The below diagram shows technological capabilities and organizational constructs.

In the lab environment we’ll install all solutions as single node instances with the following sizings:

NamePurposeSizevCPUMemory (GB)Disk (GB)
vrslcm1Lifecycle Manager2678
wsa1avIDMMedium81660
vra1avRealize AutomationMedium1242236
vrops1avRealize OperationsExtra small28274
vrli1avRealize Log InsightSmall48530

After the deployment of these solutions, we’re going to initially integrate them.

Page 9 of 12

All your base are belong to us.