If you’ve moved to VCF Automation 9 and gone looking for the old Aria Automation 8.x style standalone appliance you used to SSH into, you’ll have noticed that it is now a fleet-level component on a dedicated VCF services runtime, lifecycle-managed by VCF Operations. And the product does substantially more than either predecessor, i.e. it merged Aria Automation’s automation/self-service with VMware Cloud Director’s tenancy, and added cloud-native (VKS/container) consumption. Which is a lot of moving parts consolidated into one place – and it is not obvious where everything now runs. So I sat down and mapped it out in this short blog post.
How VCF Automation 9.1 is put together
VCF Automation (VCFA) consists of one or more so-called VMSP (VMware Management Services Platform) appliances which run containerized services in a Kubernetes cluster. Such an appliance has a significant resource footprint (24 vCPUs, 96 GB RAM) as it combines all the services that were previously running in separate appliances.
It helps to know where this thing comes from. VCFA is VCF 9’s new Cloud Management Platform, and it supersedes both Aria Automation and VMware Cloud Director (VCD) – the two products many of us ran separately for years. VCFA is a genuinely new solution first introduced in VCF 9.0, not a rebadge of an old one. It just reuses a lot of its predecessors’ parts. Roughly speaking it has two halves. The provider side – aka the Tenant Manager – is built on VCD code, so its UI and APIs feel immediately familiar if you’ve run VCD. The tenant side inherits heavily from Aria Automation and will look brand new to VCD end-users. That split is worth holding onto, because it explains why some of what you’ll meet below reads like an old friend and some of it doesn’t.
Underneath it sits the VCF Services Runtime, and that’s the piece that actually runs Automation as containerised workloads. It’s a purpose-built Kubernetes cluster built from VM nodes that each carry both control-plane and worker roles – there’s no separate master/worker split to think about. Two things are worth knowing up front: VCF Automation 9.1 gets its own dedicated services runtime instance, separate from the runtime that VCF Management Services uses, and that runtime needs its own dedicated FQDN and IP for internal communication between VCF components, kept distinct from the IP pool handed to the Automation nodes.
The addressing is where it’s easy to trip up, because there are three separate things in play:
- VCF Automation FQDN: The address of the Consumption and User Interface itself, i.e. the UI and API your users and tenants hit.
- VCF Services Runtime FQDN: The runtime’s own dedicated address for internal component communication. This is the box we usually log into using SSH.
- Services Runtime IP Pool: 5 addresses handed to the Automation node VMs, kept deliberately separate from the runtime FQDN above.
Those five addresses aren’t a round number by accident. A VCF Automation deployment wants a dedicated pool of exactly five: three for the active nodes, and two held back as a buffer to allow the platform to perform automated rolling upgrades, node replacements (self-healing), or seamless scale-out actions (i.e. moving from a Simple deployment to a 3-node HA deployment) without reconfiguring network subnets. That buffer is what makes the two deployment shapes possible:
| Component | Simple Deployment | HA Deployment |
| Node Count | 1 node VM | 3 active node VMs |
| Services Runtime IP Pool | 5 IP addresses (3 assigned, 2 buffer) | 5 IP addresses (3 assigned, 2 buffer) |
| Automation Virtual IP | 1 FQDN / 1 IP Address | 1 FQDN / 1 IP Address |
| Dedicated Runtime Management IP | 1 FQDN / 1 IP Address | 1 FQDN / 1 IP Address |
| Application-level Resiliency | No application-level HA | High Availability across 3-node cluster |
| Update / Patch Window Service Uptime | Brief service unavailability during updates | Non-disruptive, uninterrupted service during updates |
The following diagram depicts the two deployment models.
The nodes themselves are managed dynamically through Cluster API (CAPI) rather than being hand-placed, and the whole runtime is pinned to a specific vCenter, cluster, resource pool, datastore and network – handy to know when you’re working out where these VMs physically live.
So when someone says “the service runtime,” they mean that Kubernetes cluster on the services runtime node(s). All of Automation runs as pods on top of it, which is why the runtime is such a chunky VM – you’re not running one Java app, you’re running an entire microservice estate plus its databases and message brokers. Internal management services (fleet-management webhooks, health-check APIs and the like) talk straight to the Kubernetes control plane to carry out day-2 work: patching, scaling, and self-healing.
The following diagram depicts the VCFA architecture.
The pod zoo
Now let’s inspect the services runtime in more detail. Getting on it is the same regardless of deployment mode: SSH to the services runtime FQDN (in my lab vcfa-sr01.vcf.sddc.lab) as vmware-system-user:
ssh vmware-system-user@vcfa-sr01.vcf.sddc.lab
sudo -i
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get pods -n preludeEverything Automation-related lives in the prelude namespace. That name is a leftover from the Aria Automation days – if you worked with Aria Automation, it’ll look familiar.
There are somewhere north of sixty pods in there. Here’s the actual output from the service runtime, unedited:
# kubectl get pods -n prelude
NAME READY STATUS RESTARTS AGE
abx-service-app-59cdf4b697-t2nwc 2/2 Running 0 40d
account-manager-server-7674df4f6b-nxpz4 1/1 Running 3 (18d ago) 40d
agent-gateway-server-5648596ccb-xhnfw 1/1 Running 0 40d
api-gateway-server-78d68cd6fd-w54dz 1/1 Running 0 38m
approval-service-app-85fb49777d-dlzqt 1/1 Running 0 40d
authentication-server-847c58948d-wfhg8 1/1 Running 3 (18d ago) 40d
catalog-service-app-678f489fd6-4kg4k 1/1 Running 0 40d
ccs-avi-eas-app-575566c89f-g29qx 1/1 Running 0 40d
ccs-gateway-app-85b5d86fd-sdnmd 1/1 Running 0 40d
ccs-infra-eas-app-756d6d4ddd-nx7wk 1/1 Running 0 40d
ccs-k3s-app-7f78bfc49d-p2pps 2/2 Running 3 (18d ago) 40d
ccs-k3s-post-install-job-psxzl 0/1 Completed 0 40d
ccs-nsx-eas-app-79d449bc9-pjj8t 1/1 Running 0 40d
ccs-vksm-eas-fb95ff7b9-v2d25 1/1 Running 0 40d
cgs-service-app-577bccd996-2xz8n 1/1 Running 0 40d
cloud-automation-ui-app-54d4bb8f8c-gmchm 1/1 Running 0 40d
cluster-object-service-server-6b4857566d-txp9g 1/1 Running 3 (18d ago) 40d
cluster-reaper-server-897d64b9b-4g6fp 0/1 Pending 0 38m
cluster-service-server-774c9994fb-r95zl 1/1 Running 3 (18d ago) 40d
cluster-sync-egest-8675fbb5d-6tgkb 1/1 Running 0 40d
cluster-sync-ingest-69d9677f4d-4nzpl 1/1 Running 0 40d
csp-fixture-job-v9lgc 0/1 Completed 0 40d
dataprotection-server-68989899d-2f8tp 1/1 Running 3 (18d ago) 40d
disable-health-status-external-http-29736660-tm2wh 0/1 Completed 0 40d
ebs-app-665476bb7f-bvd2r 1/1 Running 0 40d
encryption-manager-5778b9fd99-r5kdn 1/1 Running 3 (18d ago) 40d
extensibility-ui-app-6df8768fb9-zhxrz 1/1 Running 0 40d
feature-flag-service-server-7cfbcbb7c8-lk2bc 1/1 Running 0 40d
hcmp-service-app-7b6dbbfd6f-ddwbt 1/1 Running 0 40d
health-status-app-f8cck 2/2 Running 0 40d
health-status-hooks-post-lbrs5 0/1 Completed 0 40d
health-status-hooks-pre-6gvsq 0/1 Completed 0 40d
intent-server-58ff8cb48c-ctrpr 1/1 Running 3 (18d ago) 40d
km-ui-server-6c79894cd6-bv9t4 1/1 Running 0 40d
nginx-httpd-app-7689f94dfd-fq486 1/1 Running 0 40d
orchestration-ui-app-f568cc46f-hwxdk 1/1 Running 0 40d
package-manager-server-6d594f866-pb6k8 1/1 Running 0 40d
policy-engine-server-68c5dddd9c-rt4hx 1/1 Running 3 (18d ago) 40d
policy-insights-server-f95f7465f-j8z5z 0/1 Pending 0 38m
policy-sync-service-server-68c58b8559-sgtcg 1/1 Running 0 40d
policy-view-service-server-57969f5c7c-vk8v9 1/1 Running 0 40d
provisioning-service-app-8586b5d7d-srknc 1/1 Running 0 40d
provisioning-service-db-upgrade-gpmmq 0/1 Completed 0 40d
provisioning-ui-app-55cfd6c957-9hq8s 1/1 Running 0 40d
proxy-service-fc44b9f85-djglv 1/1 Running 0 40d
rabbitmq-ha-0 1/1 Running 0 40d
relocation-service-app-645df95548-89dh4 1/1 Running 0 40d
relocation-ui-app-57fd6b47b6-w7wvh 1/1 Running 0 40d
resource-manager-server-7865d5ccfb-rv5vz 1/1 Running 4 (18d ago) 40d
svc-logrotate-fw96s 1/1 Running 0 40d
tango-blueprint-service-app-58779f8887-p8vfz 1/1 Running 0 40d
tango-uber-service-app-c8fdb545c-qbnzs 1/1 Running 0 40d
tenant-manager-0 1/1 Running 0 40d
terraform-service-app-79dd957595-p6qc8 2/2 Running 0 40d
vcfa-common-hooks-post-install-mvm6l 0/1 Completed 0 40d
vcfa-common-hooks-prerequisites-k4v9w 0/1 Completed 0 40d
vcfa-service-manager-665d87ff4-dv5r4 2/2 Running 3 (18d ago) 40d
vcfapostgres-0 3/3 Running 0 40d
vcfapostgres-pooler-67cf4648db-fsqtl 1/1 Running 0 40d
vcfapostgres-pooler-67cf4648db-gdr7p 1/1 Running 0 40d
vco-app-0 2/2 Running 1 (33d ago) 40d
vksm-kafka-cluster-cruise-control-87dfc6744-msjxd 1/1 Running 0 8m20s
vksm-kafka-cluster-entity-operator-5945459768-94v8b 2/2 Running 0 38m
vksm-kafka-cluster-kafka-exporter-79f59cd775-9tn4p 1/1 Running 0 8m20s
vksm-kafka-cluster-vksm-kafka-cluster-broker-0 1/1 Running 0 13h
vksm-kafka-cluster-vksm-kafka-cluster-broker-1 1/1 Running 0 18h
vksm-kafka-cluster-vksm-kafka-cluster-broker-2 1/1 Running 0 14h
vksm-kafka-cluster-vksm-kafka-cluster-controller-3 1/1 Running 0 19h
vksm-kafka-cluster-vksm-kafka-cluster-controller-4 1/1 Running 0 20h
vksm-kafka-cluster-vksm-kafka-cluster-controller-5 1/1 Running 0 21hIt looks intimidating at first glance, but the pods group cleanly by job. Here’s how I carve them up.
The front door, i.e. how requests get in, and who you are:
api-gateway-server: the main API entry point for everythingnginx-httpd-app: reverse proxy and web front endproxy-service: internal request routingauthentication-server: the authentication service (this is the login path)account-manager-server: users and accountsagent-gateway-server: the inbound endpoint remote agents phone home to
The provider control plane:
tenant-manager-0: the provider/tenancy control planevcfa-service-manager: looks after the lifecycle of the Automation services themselves
Self-service and the UIs:
catalog-service-app: the service catalogcloud-automation-ui-app: the main VCFA UIprovisioning-ui-app,orchestration-ui-app,extensibility-ui-app,relocation-ui-app,km-ui-server(+): the purpose-built front-ends for each areaapproval-service-app: approval policies
Provisioning and blueprints — the classic engine:
provisioning-service-app— the workhorse that actually builds thingstango-blueprint-service-app,tango-uber-service-app— the blueprint services (tango is the old Cloud Assembly codename)terraform-service-app: Terraform-backed deploymentsresource-manager-server,package-manager-server: track resources and content
Extensibility:
ebs-app: the Event Broker Serviceabx-service-app: Action-Based Extensibility, the serverless-actions bitvco-app-0: the embedded Orchestrator; if you’ve ever written a vRO workflow, that’s your old friend running as a pod
Policy and governance:
policy-engine-server,policy-sync-service-server,policy-view-service-server,policy-insights-server: the policy stackcgs-service-app(+),hcmp-service-app(+): I’m fairly sure these sit in the governance / management-platform space…
Cloud Consumption Services (CCS) – the newer Kubernetes-native, “All Apps” side, basically a little world of its own:
ccs-gateway-app: fronts CCSccs-k3s-app: a nested k3s cluster running inside the runtime (yes, Kubernetes inside Kubernetes)ccs-avi-eas-app,ccs-infra-eas-app,ccs-nsx-eas-app,ccs-vksm-eas: adapter pods out to load balancing, compute, networking and the Kubernetes service (I read EAS as “external adapter service”)cluster-object-service-server,cluster-service-server,cluster-reaper-server,intent-server,cluster-sync-ingest,cluster-sync-egest: the declarative machinery that reconciles what you asked for down onto the vSphere Supervisors
The Kafka block:
vksm-kafka-cluster-*: a Strimzi-managed Kafka cluster (three brokers, three controllers, plus entity-operator, exporter and cruise-control), the event bus behind the vSphere Kubernetes Service manager. In this capture they’ve got the youngest ages on the list, which means they’d been restarted or rescheduled shortly before I grabbed the output.
Data and messaging:
vcfapostgres-0: the platform Postgres (it shows 3/3 containers because it’s a proper HA setup)vcfapostgres-pooler: the connection pooler in front of itrabbitmq-ha-0: the message broker the older services lean on
Housekeeping:
dataprotection-server: backupencryption-manager: secretsrelocation-service-app: migrations off vRA 8feature-flag-service-server,health-status-app,svc-logrotate: feature flags, health checks, log rotation- the various
*-hooks-*and*-jobpods: one-shot install/upgrade jobs
A quick disclaimer on the descriptions above: the pod list is straight off my lab appliance, but the “what each one does” is my own reading, part from experience with the Aria Automation lineage these names came from, part educated guess. I’ve flagged the ones I’m unsure about with (+). Don’t go looking for a neat official table that names all of these either, because as far as I can tell it doesn’t exist.
That’s the map and that’s all for this time 🙂
Leave a Reply