If you run VMware Cloud Foundation 9.1 nested in a homelab, VCF Automation (VCFA) is the appliance that will make your host fans spin. It ships as a single “one size fits all” deployment sized at a minimum of 24 vCPU + 96 GB RAM, and internally it’s a full Kubernetes platform. Drop that onto a modest nested host and you’ll see high CPU, contention, and – if you start “optimizing” the wrong things like me – a spectacular cascade of failures. In this blog I’ll describe my journey to get a stable and performing VCFA appliance in my homelab.
In my homelab I’m running VCF 9.1 plus VCFA on a single-socket Intel Xeon Gold 5412U (24 cores) host with 512 GB of physical memory. You can check the details of my setup here…
Like probably most people, I’ve started tweaking performance for bringing the appliance’s footprint down by applying community optimization guides. In particular, I tried Tom Fojta’s Downscaling VCF Automation 9.1 (shrink the internal Kafka cluster). While trying to implement this, I learned what genuinely relieves the pressure, what looks like an optimization but is a trap, and how to recover when it goes sideways. Warning: All of it is unsupported homelab territory – we’d never apply it to production!
First, diagnose
“High CPU” has two completely different causes, and the fixes are opposite. On the host, esxtop and typing c tells us whether we’re contended (high %CSTP, PCPU not pegged means too-wide VM) or genuinely saturated (PCPU near 100% means real demand). VMware’s Performance Best Practices for vSphere 9.0 explains why oversized wide VMs hurt scheduling.
Inside the VCFA appliance, we check CPU requests and the pod count – the second is the one everyone forgets. The node’s kubelet has a hard pod cap:
root@vcfa-sr01-2p28v [ ~ ]# ps -ef | grep kubelet | grep -v grep | tr ' ' '\n' | grep max-pods
--max-pods=224Remember that 224. It turned out to be our real ceiling – not the cores.
Right-size the appliance
A 24-vCPU VM on a 24-core socket is the worst possible shape for the scheduler: for that VM to run, ESXi has to co-schedule all 24 vCPUs nearly simultaneously, evicting everything else off the physical cores at once. I saw it as high %CSTP across a lot of my VMs, not just VCFA. The fix is to make the appliance narrower than the socket.
After a lot of testing, I’ve dropped it to 20 vCPU.
Why not 16 as stated in some community guides? Because 16 is below my particular appliance’s actual footprint. Let me explain.
VCFA sits between two limits: too wide and the hypervisor can’t schedule the VM; too narrow and Kubernetes inside the VM can’t fit its own pods. 20 vCPU is the number that satisfies both.
Too wide (24): hypervisor co-scheduling. vSphere runs a VM’s vCPUs as a group, so a 24-vCPU VM on a 24-core socket can only run when the entire socket is free – every time it needs CPU it effectively shoves all other VMs off the cores at once. That shows up as high %CSTP in esxtop across all your VMs, even though the host isn’t out of raw compute. Any width below the socket relieves it.
Too narrow (16): Kubernetes can’t place its pods. The trap is “it barely uses 24, so shrink it hard”. But Kubernetes schedules on reservations, not usage: every pod requests (reserves) some CPU, and a pod only starts if its reservation fits… no matter how idle the running pods actually are. On my VCFA those reservations sum to ~17 vCPU, so 16 simply can’t seat them all. At 16 the node’s own accounting showed reservations already at 97%:
root@vcfa-sr01-2p28v [ ~ ]# kubectl describe node | grep -A6 "Allocated resources"
Allocated resources:
Resource Requests Limits
cpu 15555m (97%) 42520m (267%)
memory 69426994Ki (76%) 154286734592 (166%)
root@vcfa-sr01-2p28v [ ~ ]# kubectl describe node | grep -iE "cpu:"
cpu: 16
cpu: 15890mCPU is in millicores (1000m = 1 vCPU), so 15555m ≈ 15.5 vCPU reserved. Only the Requests column gates scheduling and must stay under 100%; Limits (267%) is just how high pods may burst if CPU is free, and is expected to exceed 100%. That 97% actually understated it as it excludes Pending pods, and at the time the Kafka controller, broker-2, and the entity-operator were all Pending (another ~1.5–2 vCPU). This overflow pushed the Kafka controller and brokers straight into unschedulable with 16 vCPUs – the below snippet shows the broker that couldn’t get a slot:
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n prelude describe pod vksm-kafka-cluster-vksm-kafka-cluster-broker-2 | grep -A4 Events
Warning FailedScheduling 0/1 nodes are available: 1 Insufficient cpu.Here broker-2 is the pod we captured the event for, but the same Insufficient cpu verdict falls on any pod that can’t fit – and the one that matters most is the KRaft controller. It’s not an optional extra: a Pending controller takes Kafka NotReady, and the whole app tier with it.
So 16 vCPU isn’t a slower appliance, it’s a broken one in my case – the pods that make it work can’t be placed!
Just right (20): fits with headroom. 20 is below the socket (no hypervisor contention) and comfortably above the ~17 vCPU of reservations. The same requests that overflowed at 16 now sit at 85%:
root@vcfa-sr01-2p28v [ ~ ]# kubectl describe node | grep -A6 "Allocated resources"
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 17065m (85%) 47020m (236%)
memory 72032050Ki (79%) 160594967808 (173%)
root@vcfa-sr01-2p28v [ ~ ]# kubectl describe node | grep -iE "cpu:"
cpu: 20
cpu: 19880mThat 17065m is the approx. 17 vCPU footprint, i.e. Kubernetes’ own sum of every scheduled pod’s reservation. It fits in 20; it can’t fit in 16. And note it reads higher than the 16-vCPU figure (15555m) – not because demand grew, but because that Requests total only sums scheduled pods: at 16 the node couldn’t place everything, so the ~1.5 vCPU of Pending pods weren’t in the sum. A too-small node reports fewer requested millicores because it drops pods on the floor, not because it needs less – the lower number is the symptom, not reassurance.
My verdict: Scale down to 20 vCPU to have enough headroom for a standard VCFA pod setup. Lower values are possible only at the cost of downscaling application availability (only do it if you have a physical core constraint).
Resize the Kafka cluster
A community guide shows shrinking the internal VKSM Kafka cluster from 3+3 to 1+1 to reclaim 4 vCPUs. On my appliance which was already running for some time, it was not possible to resize exactly as described. The proposed edit is:
kubectl -n prelude edit kafkacluster vksm-kafka-cluster
...
spec:
broker:
replicas: 1
storage:
size: 22Gi
controller:
replicas: 1
storage:
size: 4Gi
...But the brokers won’t actually reduce, the Strimzi operator reverts them:
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n prelude get kafkacluster vksm-kafka-cluster \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
NotReady=False Reverting scale-down of KafkaNodePool vksm-kafka-cluster-broker by changing number of replicas to 3
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n prelude get kafkanodepool
NAME DESIRED REPLICAS ROLES NODEIDS
vksm-kafka-cluster-broker 1 ["broker"] [0,1,2]
vksm-kafka-cluster-controller 1 ["controller"] [3] Those two node pools are the two Kafka roles, and the ROLES column names them.
The brokers (["broker"], IDs 0–2) hold the actual topic data. The controller (["controller"], ID 3) is the KRaft controller already mentioned earlier, i.e. the node that manages the cluster’s metadata. Note: KRaft is Kafka’s ZooKeeper-free mode, mandatory in Kafka 4.x, which this cluster runs; the controller quorum is the source of truth for the entire cluster, and if it’s unavailable Kafka can’t function at all.
Hold onto that distinction: brokers and controllers behave completely differently when you try to shrink them.
Note the mismatch: DESIRED REPLICAS 1, but NODEIDS [0,1,2] and three running brokers. The operator forces 3 because the internal topics need the replicas – our user topics are RF=1, but the internal ones aren’t:
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n prelude get kafkatopic \
-o custom-columns=NAME:.metadata.name,PART:.spec.partitions,RF:.spec.replicas
NAME PART RF
cluster-sync-kafka-topic 3 1
cluster-sync-reaper-clusters 1 1The real gate is Strimzi’s broker scale-down check: before deleting a broker, the operator verifies that broker holds no partition replicas – if it does, removing it would lose data, so it reverts the count (the message above).
That also explains why the very same edit worked in the community post but not for me. His broker.replicas: 1 almost certainly landed on a fresh deployment, where brokers 1 and 2 were still empty and the check passed; my 40-day old appliance has partitions on all three brokers, so the check fails and reverts. To force it we’d have to reassign every partition off brokers 1 and 2 onto the surviving broker first (and since the topics are RF=1, there’s no second copy as a safety net), or bypass the check with the strimzi.io/skip-broker-scaledown-check: "true" annotation – which risks losing the Kafka data the appliance runs on. Not worth ~2 vCPU. So I left the brokers at 3 and just set the spec back to match, which stops the reconcile churn:
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n prelude edit kafkacluster vksm-kafka-cluster
...
spec:
broker:
replicas: 3
storage:
size: 22Gi
controller:
replicas: 1
storage:
size: 4Gi
...Let’s check:
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n prelude get kafkanodepool
NAME DESIRED REPLICAS ROLES NODEIDS
vksm-kafka-cluster-broker 3 ["broker"] [0,1,2]
vksm-kafka-cluster-controller 1 ["controller"] [3]Unlike the brokers, the operator let controller.replicas: 1 stand, so our quorum dropped from the shipped 3 to the single controller-3 you saw in the node pool above. A lone controller is a single point of failure for the cluster metadata: during our ordeal, when that one controller couldn’t schedule (the pod-cap problem below), there was no controller running at all, and the whole Kafka layer went NotReady:
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n prelude get kafkacluster vksm-kafka-cluster -o yaml | sed -n '/status:/,$p'
status:
conditions:
- message: 'org.apache.kafka.common.errors.TimeoutException: Timed out waiting for
a node assignment. Call: listTopics'
reason: CompletionException
status: "False"
type: NotReady
kafkaVersion: 4.1.0That’s the core warning of this section: cutting the controllers from 3 to 1 – half of what the 1+1 downscale does – is what turned a recoverable pod-cap hiccup into a full metadata outage.
My verdict: aim for a 3 brokers + 1 controllers setup if you have more than 16 physical cores and a freshly installed system. The broker downscale was not possible as the operator reverts it on a longer running system. Keep in mind that reducing the controllers is a single point of failure, but it is acceptable if the single host doesn’t have the headroom the shipped 3 assume..
Full disclosure: My appliance ran at 3 brokers + 1 controller for a while because I thought I was pinned near the pod cap – but once the support-bundle leak (see next section) was fixed, the pod count dropped to ~143 / 224, plenty of room. So I restored the controllers to the shipped 3; on this footprint it adds ~0.9 vCPU, taking requests to ~91% at 20 vCPU – the requests still fit (~91% at 20 vCPU). But requests fitting isn’t the same as running. On my single host with CPU steal, the extra load pushed the run queue past what the node could turn over and etcd starved, i.e. resulted in errors such as apply request took too long, waiting for ReadIndex response took too long, the apiserver losing etcd (dial tcp 127.0.0.1:2379: connection refused) – and the control-plane VIP dropping. The API went fully dark (Unable to connect to the server: ... no route to host). This was starvation, not corruption (no mvcc/panic/checksum errors). So my final fix was to drop the controller replicas back to 1.
Leave the vCPU at 20 either way: ~17 vCPU of requests is the fixed floor, and 16 doesn’t fit regardless of how much pod headroom you have.
The real killer: pod-count exhaustion and a leaking cron job
During all the above investigations, I discovered what actually took my VCFA appliance down. The kubelet caps the node at --max-pods=224 (shown earlier). VCFA’s healthy footprint already runs close to that, and a built-in platform CronJob pushes it over. After a reboot, our pod count sat at 312 – a wall of Unknown ghosts:
root@vcfa-sr01-2p28v [ ~ ]# kubectl get pods -A --no-headers | awk '{print $4}' | sort | uniq -c | sort -rn
221 Running
166 Unknown
25 Completed
5 Pending
5 OutOfpods
...
root@vcfa-sr01-2p28v [ ~ ]# kubectl get pods -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -rn | head
336 vmsp-platform
69 prelude
13 kube-systemNearly all of vmsp-platform‘s pods were Unknown, and once the node hit maxPods, new pods – including the Kafka controller and the app tier – went unschedulable with Too many pods:
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n prelude describe pod vksm-kafka-cluster-vksm-kafka-cluster-broker-2 | grep -A4 Events
Warning FailedScheduling ... 0/1 nodes are available: 1 Too many pods. ...The culprit I found is support-bundle-cluster-info-dump – a slow full-cluster export scheduled every 12h under concurrencyPolicy: Allow. It never finishes before the next run starts, reboots orphan in-flight runs, and they stack! The ACTIVE count told the whole story – 82 jobs still active:
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n vmsp-platform get cronjob | grep support-bundle
support-bundle-cluster-info-dump 0 */12 * * * <none> False 82 7h59m 41d
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n vmsp-platform get cronjob support-bundle-cluster-info-dump \
-o jsonpath='schedule={.spec.schedule} succHist={.spec.successfulJobsHistoryLimit} failHist={.spec.failedJobsHistoryLimit} suspend={.spec.suspend} concurrency={.spec.concurrencyPolicy}{"\n"}'
schedule=0 */12 * * * succHist=3 failHist=1 suspend=false concurrency=AllowThe job wasn’t crashing, but its log shows it grinding through an exhaustive object export, one CRD type per namespace, self-throttled:
root@vcfa-sr01-2p28v [ ~ ]# job=$(kubectl -n vmsp-platform get jobs --no-headers \
| awk '$1 ~ /^support-bundle-cluster-info-dump/{print $1; exit}')
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n vmsp-platform logs job/$job --tail=5
INFO cluster-info-dump 1 Exporting objects: type=flows.logging.banzaicloud.io, scope=prelude
INFO cluster-info-dump 1 Exporting objects: type=outputs.logging.banzaicloud.io, scope=kube-system
...I fixed it at the source. i.e Forbid so runs can’t stack, suspend it, and delete the backlog:
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n vmsp-platform patch cronjob support-bundle-cluster-info-dump \
--type merge -p '{"spec":{"concurrencyPolicy":"Forbid","suspend":true}}'
cronjob.batch/support-bundle-cluster-info-dump patched
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n vmsp-platform get jobs --no-headers \
| awk '$1 ~ /^support-bundle-cluster-info-dump/{print $1}' \
| xargs -r -n1 kubectl -n vmsp-platform delete job --wait=false
job.batch "support-bundle-cluster-info-dump-29747520" deleted
... (82 of these) ...
root@vcfa-sr01-2p28v [ ~ ]# kubectl -n vmsp-platform get cronjob support-bundle-cluster-info-dump
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
support-bundle-cluster-info-dump 0 */12 * * * True 0 8h 41d
root@vcfa-sr01-2p28v [ ~ ]# kubectl get pods -A --no-headers | grep -vE 'Completed|Succeeded' | wc -l
223That reclaimed round-about 80 pod slots and dropped the node back under the cap.
The Unknown pods above accumulate on every reboot, i.e. the kubelet loses track of pre-reboot containers and never reaps them. They display as Unknown but their .status.phase is not Unknown, so a phase field-selector misses them; I match the STATUS column instead. So I’ll probably make this a post-reboot habit:
root@vcfa-sr01-2p28v [ ~ ]# kubectl get pods -A --no-headers | awk '$4 ~ /Unknown/{print $1, $2}' \
| while read ns pod; do kubectl -n "$ns" delete pod "$pod" --force --grace-period=0; done
root@vcfa-sr01-2p28v [ ~ ]# kubectl get pods -A --field-selector=status.phase=Succeeded --no-headers \
| awk '{print $1, $2}' | while read ns pod; do kubectl -n "$ns" delete pod "$pod" --force --grace-period=0; done
root@vcfa-sr01-2p28v [ ~ ]# kubectl get pods -A --no-headers | grep -vE 'Completed|Succeeded' | wc -l
223My verdict: this is a serious platform defect/bug in my testing (unbounded collector under concurrencyPolicy: Allow, no activeDeadlineSeconds, weak pod GC).
Conclusion
My biggest mistake was shrinking the VM before reducing demand – which is how a tuning exercise became a multi-hour outage. The correct order, always:
- Diagnose both axes: CPU (
esxtop %CSTPvsPCPU,describe nodeallocations) and pod count (maxPodsvs actual). Pod-count exhaustion masquerades as a CPU problem. - Fix the pod-cap leak (suspend the CronJob, clear ghosts) so you have headroom.
- Reduce demand before capacity: trim requests first (never the Kafka controllers), verify Ready + under the pod cap, then resize the VM.
- Resize gracefully to 20 vCPU for a default setup.
- Verify end-to-end: not just green pods, but an actual UI login.
The decisive do / don’t for my homelab setup:
Do:
- Run the appliance at 20 vCPU (narrower than the socket, above the ~17 vCPU footprint).
- Keep Kafka at 3 brokers + 1 controllers, and fix the pod cap so that one controller always schedules.
- Suspend the
support-bundle-cluster-info-dumpCronJob (and setconcurrencyPolicy: Forbid). - Clear
Unknownpods after every reboot. - Shut down gracefully via VCF Operations before any resize.
Don’t:
- Don’t run it at 24 vCPU (full-socket co-scheduling contention).
- Don’t shrink to 16 vCPU (requests hit 97% and overflow – we measured it) unless you’re on a freshly installed VCFA appliance.
- Don’t restore the Kafka controllers to 3 on a single-host lab – the extra load starved our etcd and dropped the control-plane VIP.
- Don’t shrink the VM before reducing demand.
- Don’t hard-power-off a Kubernetes appliance.
The honest headline from this session: the effective performance fix for VCFA 9.1 in a homelab is right-sizing the VM to 20 vCPU and stopping a leaking CronJob! CPU was never our ceiling; the maxPods=224 pod cap was. Fix the right thing and the appliance is stable, healthy, and far kinder to your host 🙂
Leave a Reply