In my previous blog post, I mapped out the service runtime and the army of pods that make up the prelude namespace in VCF Automation 9.1, and covered how to get a shell on the runtime. Today I put that to work on a real problem, and dig into one of the most interesting pods in the namespace.

Recently, I had a customer, where I needed to get detailed logs why a user login into a VCFA tenant organization didn’t work. There’s no button in the UI for “explain why this login failed,” so the only way forward was to get onto the box that runs Automation, find the right service, and read its logs. That service is the so-called tenant-manager.

tenant-manager is a core microservice in the prelude namespace, and on the provider side it’s the control plane for multi-tenancy, i.e. the engine that carves a shared private cloud up into isolated consumer organizations. Concretely it’s doing the following jobs:

  • Multi-tenancy and isolation: creating, isolating and managing independent organizations and their tenant boundaries inside shared infrastructure.
  • Resource allocation and quotas: managing Region Quotas and handing compute, memory, storage and network quotas to each organization, wiring down into vSphere Supervisor Namespaces and NSX constructs so one tenant can’t starve another (the classic noisy-neighbour problem).
  • Identity and access: tenant-scoped identity management and RBAC. It resolves users and group mappings from enterprise identity providers (OIDC, vIDM) so tenant users land in the projects and namespaces they’re entitled to, and it looks after the SSL certificate trusts behind identity federation.
  • Tenant notifications: brokering notifications from operational tools (VCF Operations alerts, Region Quota thresholds) out to the right tenant administrators.

Keep that identity-and-access line in mind, because it’s the reason a “one organization can’t log in” ticket was always going to end up in this pod.

First, we need to login to the VCFA Services Runtime control plane node using SSH:

ssh vmware-system-user@vcfa-sr01.vcf.sddc.lab
sudo -i
export KUBECONFIG=/etc/kubernetes/admin.conf

If you’re unsure about the services runtime FQDN, you can look it up in VCF Operations under Build > Lifecycle > VCF Management > VCF Automation, as shown in the screenshot below (check the VCF services runtime FQDN entry):

Now, let’s get the tenant-manager pod:

kubectl get pods -n prelude | grep tenant-manager

It reveals the following:

tenant-manager-0                        1/1      Running      45  (2d7h ago)        31d

Exec into this pod:

kubectl exec -it tenant-manager-0 -n prelude -- /bin/bash

And we land at a shell prompt that reads vcloud [ /opt/vmware ]. The user is vcloud, the logs we’re searching for live under /opt/vmware/vcloud-director/logs/.

In this directory, we find the vcloud-container-debug.log which is a firehose and well known to all VCD admins. The snippet below shows the failed authentication attempt and, more importantly, the error that points us towards the actual root cause…

2026-08-24 11:14:31,010 | DEBUG    | pool-jetty-1893760        | c.v.s.b.o.OIDCServiceImpl      | Failed to retrieve OIDC user information | requestTime=1787656470563, request=GET https://flt-auto01.vcf.sddc.lab/login/oauth, requestInfo=requestId=185e412e-3bbe-4167-a065-cfb489e087ac,request=GET https://flt-auto01.vcf.sddc.lab/login/oauth,requestTime=1787656470563,remoteAddress=192.168.123.198:48654,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=text/html application/xhtml+xml application/xml;q 0.9 image/avif image/webp image/apng */*;q 0.8 application/signed-exchange;...,Host=flt-auto01.vcf.sddc.lab, userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ..., Host=flt-auto01.vcf.sddc.lab, REQUEST_ID=185e412e-3bbe-4167-a065-cfb489e087ac, remoteAddress=192.168.123.198:48654, accept=text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
java.lang.IllegalArgumentException: Attribute value for 'preferred_username' cannot be null
        at org.springframework.util.Assert.notNull(Assert.java:181)

Simple as that 🙂