In my previous blog post, I mapped out the service runtime and the army of pods that make up the prelude namespace in VCF Automation 9.1, and covered how to get a shell on the runtime. Today I put that to work on a real problem, and dig into one of the most interesting pods in the namespace.

Recently, I had a customer, where I needed to get detailed logs why a user login into a VCFA tenant organization didn’t work. There’s no button in the UI for “explain why this login failed,” so the only way forward was to get onto the box that runs Automation, find the right service, and read its logs. That service is the so-called tenant-manager.

tenant-manager is a core microservice in the prelude namespace, and on the provider side it’s the control plane for multi-tenancy, i.e. the engine that carves a shared private cloud up into isolated consumer organizations. If you’re enterprise IT or a service provider, this is the piece that lets you act as an internal cloud broker: you draw the tenant boundaries and administrative domains, and it keeps them apart. Concretely it’s doing four jobs:

  • Multi-tenancy and isolation: creating, isolating and managing independent organizations and their tenant boundaries inside shared infrastructure.
  • Resource allocation and quotas: managing Region Quotas and handing compute, memory, storage and network quotas to each organization, wiring down into vSphere Supervisor Namespaces and NSX constructs so one tenant can’t starve another (the classic noisy-neighbour problem).
  • Identity and access: tenant-scoped identity management and RBAC. It resolves users and group mappings from enterprise identity providers (OIDC, vIDM) so tenant users land in the projects and namespaces they’re entitled to, and it looks after the SSL certificate trusts behind identity federation.
  • Tenant notifications: brokering notifications from operational tools (VCF Operations alerts, Region Quota thresholds) out to the right tenant administrators.

Keep that identity-and-access line in mind, because it’s the reason a “one organization can’t log in” ticket was always going to end up in this pod.

First, we need to login to the VCFA Services Runtime control plane node using SSH:

ssh vmware-system-user@vcfa-sr01.vcf.sddc.lab
sudo -i
export KUBECONFIG=/etc/kubernetes/admin.conf

If you’re unsure about the services runtime FQDN, you can look it up in VCF Operations under Build > Lifecycle > VCF Management > VCF Automation, as shown in the screenshot below (check the VCF services runtime FQDN entry):

Now, let’s get the tenant-manager pod:

kubectl get pods -n prelude | grep tenant-manager

It reveals the following:

tenant-manager-0                        1/1      Running      45  (2d7h ago)        31d

Exec into this pod:

kubectl exec -it tenant-manager-0 -n prelude -- /bin/bash

And we land at a shell prompt that reads vcloud [ /opt/vmware ]. The user is vcloud, the logs we’re searching for live under /opt/vmware/vcloud-director/logs/.

Simple as that 🙂