VMware Cloud Foundation 9.1 Deployment Guide

VMware Cloud Foundation (VCF) 9.1 has been released a few weeks ago. Although I’m already a bit late to the game, I’ve finally found the time to deploy it in my lab environment. This blog post provides a step-by step guide how to prepare my lab and it provides a deep-dive guide how to deploy the complete VCF 9.1 stack on a single physical ESX host.

In my lab design, I’ll setup a single VCF instance flt01-i01 consisting only of a Management Domain m01. The Management Domain will consist of a Supervisor-enabled vSphere cluster, which will be consumed by a VCF Automation all-apps organization. For north-south communication, we will deploy a 2-node NSX Edge cluster on this vSphere cluster as well.

The Management Domain will consist of 4 nested ESX servers, running on my physical lab ESX host. Storage is provided by an NFS server running on an Ubuntu Linux virtual machine on the same physical ESX host.

The networking setup is the same as in my previous VCF deployment scenarios:

To provide routing to the NSX fabric networks of the Workload Domain, the NSX Edges will communicate over the Edge Uplink networks 1 and 2 to a pfSense router and use BGP for route distribution. Thus, I have configured a pfSense router to support BGP using the FRR package. For this lab setup, we will define the pfSense router as to be in AS 65001 and our NSX Edges to be in AS 65003.

In this lab, we’re using the network 10.230.10.0/24 as the general SDDC management network (VLAN 10), where — among other systems — our Windows Server Domain Controller resides (dc1.sddc.lab with IP address 10.230.10.4). Overall, we’re going to use the following DNS hostnames and IP addresses for our lab setup:

The NSX side is built around Edge cluster m01-ec01 (two Large edge nodes to provide enough resources for the VCF-A all-apps organization model) hosting Tier-0 gateway m01-ec01-t0-gw01 in Active/Standby mode, local ASN 65003, peering BGP with the two top-of-rack switches.

VPC connectivity uses the Centralized transit-gateway model with an external IP block 192.168.21.0/24 and a private transit-gateway block 172.16.0.0/16.

The vSphere Supervisor m01-sn01 (zone m01-zn01) runs VCF Networking with VPC, a private VPC CIDR of 172.25.0.0/16 and a service CIDR of 172.26.0.0/16, delivering vSphere Kubernetes Service (VKS) v1.32.x.

The virtual distributed switch is m01-cl01-vds01 (MTU 9000, two uplinks vmnic0/vmnic1).

Preparing the ESX servers

The ESX servers in this lab will be deployed as nested appliances on my physical lab ESX server.

For the four Management Domain ESX servers, I’ll use the following specs:

• 24 vCPUs
  • Expose hardware assisted virtualization to the guest OS
• 122 GB Memory
• 2x VMXNET3 network adapter
  • Connected to my Trunk portgroup on the physical ESX host (VLAN 4095)
• 1x VMware Paravirtual SCSI Controller
• 1x 64 GB Harddisk connected to the SCSI Controller
• VM Boot Options
  • Whether or not to enable UEFI secure boot for this VM: Disabled

Now, that we have the deployment specification for our nested ESX servers, we must install and configure them.

First, get the ESX 9.1 installer ISO from the Broadcom Support Portal — the GA release file is called VMware-VMvisor-Installer-9.1.0.0.25370933.x86_64.iso. Then, use the following procedure for each of the ESX hosts:

Mount ESX installer ISO to the ESX server you’re going to install. Follow the installation wizard as usual, just make sure to install the OS to the hard disk on the SCSI controller. Also make sure to provide a sufficient complex password for the root user.

After we’ve installed ESX, we configure the Management network and enable SSH using the ESX DCUI, e.g. for m01-esx01:

• Network Adapters: vmnic0
• VLAN: 20
• IPv4 Address: 10.230.20.211
• Subnet Mask: 255.255.255.0
• Default Gateway: 10.230.20.1
• Primary DNS Server: 10.230.10.4
• Hostname: m01-esx01.vcf.sddc.lab
• Custom DNS Suffixes: vcf.sddc.lab

Next, we connect to the ESX host as root using SSH and setup NTP:

esxcli system ntp set -s=10.230.10.4esxcli system ntp set -e=yes

Set the vSwitch MTU size to 9000 bytes:

esxcli network vswitch standard set -m 9000 -v vSwitch0

Setup the host SSL certificate:

/sbin/generate-certificates

Reboot the ESX host:

reboot

After the ESX host has been rebooted, connect to the ESX host client using a web browser and check if the new SSL certificate has been applied and shows the correct hostname.

Repeat this procedure for the remaining ESX hosts.

VCF Installer Deployment

The VMware Cloud Foundation Installer is a single virtual appliance that deploys and configures all the required VMware Cloud Foundation components. In our lab, we’re going to deploy the VCF Installer directly on the physical ESX host.

The VCF Installer is provided as a OVA file, you can download it from the Broadcom Support Portal. In the Support Portal navigate to My Downloads > VMware Cloud Foundation > VMware Cloud Foundation 9.1.0.0 > VMware Cloud Foundation Installer, the the GA release file is called VCF-SDDC-Manager-Appliance-9.1.0.0.25371088.ova. Once we’ve downloaded the OVA file, we deploy a new VCF Installer appliance from it on the physical ESX host.

In our lab, we have connected the VCF Installer VA to a network where the appliance is able to access all required external services, such as DNS and NTP, as well as the VCF Management components. Thus, we’re deploying it in our ESX/VM Management network 10.230.20.0/24 (VLAN 20) with IP address 10.230.20.9.

Once the VA has been deployed and powered on, the VCF Installer GUI can be accessed from a web browser using https://vcf9-installer.vcf.sddc.lab.

We enter the admin@local user and the password we’ve provided when we deployed the appliance and then click Log In.

VCF Installer Binary Management

Before we can deploy VMware Cloud Foundation, we must download the required binaries to the VCF Installer appliance.

Click on Depot Settings and Binary Management.

In the lab the VCF Installer appliance can connect to the Internet (either directly or through a proxy server). Thus, we will connect it to an online depot and download the binaries using an activation code generated from the BSC Portal vcf.broadcom.com.

Under Online Depot click Configure and provide the Activation Code for this Download Service ID generated in the BSC, then click Authenticate.

We are now ready to download the binaries by selecting all of them and then clicking on Download.

Deploy VMware Cloud Foundation

Now, that all binaries are download to the VCF Installer, we can start the actual deployment by clicking on Deployment Wizard > VMware Cloud Foundation.

Let’s read the wizard introduction and click Next.

Here we select “Deploy a new VCF fleet” and click Next.

We leave all Existing Component options unchecked (greenfield) and click Next.

Let’s choose the Simple deployment model and Small size, then click Next.

You review the default network options, then click Customize.

We set VM management to reuse the ESX management network and the VPC gateway to Centralized connectivity; click Next.

Here we select NFS v3 as the principal storage type and click Next.

Now we click Pre-Fill Generated FQDNs to create the management hostnames according to our naming convention. These hostnames will be later automatically prefilled in the respective hostname fields of the installer forms — very handy 🙂

Now you click Validate All, confirm every FQDN resolves (green check marks), then Apply.

Now we enter the version (9.1.0.0), instance name flt01-i01, management domain m01 and DNS/NTP 10.230.10.4; click Next.

Let’s add our ESXi hosts, enter the root passwords and confirm the SSH fingerprints.

We acknowledge the resource-requirements warning and click Proceed.

Then we define the ESX management (VLAN 20), vMotion (VLAN 30), NFS (VLAN 40) and NSX overlay (VLAN 50) networks with MTU 9000; click Next.

Next we verify the FQDNs for Operations, Cloud Proxy, License Server, VCF management services and Automation; click Next.

You verify the vCenter FQDN, datacenter name m01-dc01, cluster name m01-cl01 and SSO domain name vsphere.local; click Next.

We confirm the NFS datastore m01-cl01-ds-nfs01 (export /srv/nfs/vmw-m01 on 10.230.40.8); click Next.

You can choose Custom Switch Configuration and click Next.

For ease, we choose Copy From Preconfigured Profile and select the Default profile.

Configure vDS m01-cl01-vds01 (MTU 9000, uplinks vmnic0/vmnic1) and enable all four traffic types.

We set the port-group teaming (Route Based on Physical NIC Load) and NSX to Standard mode (Enhanced Datapath is not supported on my host); click Next.

Now you verify the NSX Manager cluster VIP and appliance name; click Next.

Now we verify the SDDC Manager FQDN; click Next.

You confirm all four ESXi hosts are added for the cluster.

We provide the relevant Network configuration as detailed in our overview chapter.

Same for the NSX overlay definitions. Then click Next.

We review the summary and click Download JSON Spec; click Next.

In my lab environment I have to acknowledge the time-sync / NFS free-space / capacity warnings, then we finally click Deploy.

Now we wait for the bring-up to finish, then log in to VCF Operations as user admin and the password we can retrieve via the Review Passwords button.

Deploy Log Management

In VCF 9.1, VCF Operations for Logs is delivered as Log Management as part of the Management Services.

In VCF Operations navigate to Build > Lifecycle > VCF Management.

Now you click ADD COMPONENT and choose Log management.

Now we enter the FQDN flt-logs01.vcf.sddc.lab, Small size and 1 replica, tick the disruption acknowledgement; click Next.

Let’s review the summary and click Finish.

We confirm the “Log management is being deployed” banner appears. After some minutes, the Log Management component has been deployed.

Deploy VCF Operations for Networks

VCF Operations for Networks (the evolution of vRealize / Aria Operations for Networks) provides network visibility, flow analysis and path troubleshooting. It is deployed as a platform node on fleet level plus one or more collector nodes.

Now we click ADD COMPONENT and choose VCF Operations for networks.

Here you select New deployment, version 9.1.0.0 and Small size; click Next.

Now we enter the admin password, platform node IP 10.230.20.81 and collector node IP 10.230.20.14; click Next.

You review the summary and click Finish.

Deploy Realtime Metrics

The real-time metrics service adds high-frequency (~20-second) telemetry to VCF Operations. Enabling it scales up the VCF services runtime cluster, so install it during a maintenance window.

Now we click ADD COMPONENT and choose Real-time metrics.

You set version 9.1.0.0 for all VCF instances, tick the acknowledgement that the services runtime scales up during a maintenance window, and click Install.

Configure Single Sign-On

VCF 9.1 centralises authentication through the VCF Identity Broker, which federates an external identity provider (here Active Directory over LDAP) to the whole fleet or instances (depending on the deployment model).

In VCF Operations, open Manage > Identity & Access > VCF SSO Overview and click CONFIGURE SSO.

We tick all prerequisite / impact acknowledgement checkboxes and click SUBMIT.

On the wizard overview, click START on Step 1 “Choose Deployment Mode”.

Here you select the Instance deployment mode, choose identity broker flt-idb01.vcf.sddc.lab, and click Next.

We confirm Step 1 shows Completed and proceed to Step 2 “Configure Identity Broker” by clicking on Start.

Under Directory-Based Identity Provider, we select AD/LDAP and click Next.

Now we click CONFIGURE on “Configure the identity provider”.

Now you enter the directory details — display name SSDC-LAB, ldap://dc1.sddc.lab:389, sAMAccountName, Base DN dc=sddc,dc=lab, bind user svc-idm-ad; click Next.

We review the directory connection and click FINISH.

Now you click CONFIGURE on “Configure user and group provisioning”.

We review the directory information and click Next.

We map the attributes and click Next.

Enable Sync Nested Groups and enter the base group DN DC=sddc,DC=lab.

Search for and select the VCF Admins group, then click Next.

We leave user provisioning empty (no individual users) and click Next.

Let’s review the selections (1 group, 0 users) and click FINISH to start the background sync.

We confirm user and group provisioning is complete (3/4). Then we click Next.

Now click TEST LOGIN.

We sign in with a user account from the VCF Admins AD group and confirm the successful login:

You confirm the green “Test login is successful” banner and click DONE.

We confirm the identity provider is configured and head over to Step 3 “Enable SSO for vCenter and NSX” by clicking on Start.

Here you select m01-vc01 (vCenter) and m01-nsx01 (NSX Manager), then click Configure.

We tick the acknowledgement that enabling SSO replaces the current vCenter authentication source, and click Finish.

You confirm Step 3 shows Completed (75%) and click Start for Step 4 “Assign VCF Roles”.

On the Groups tab, select the VCF Admins group and click ASSIGN.

You set the scope to Components with flt-idb01.vcf.sddc.lab, role to VCF Administrator, and click ASSIGN.

We confirm the role assignment succeeded and click Done.

You confirm all four steps are complete (100%), then click Finish Setup.

In the Finish Setup dialog, optionally Export Configuration, then click CONTINUE.

On the SSO dashboard, note the two management components still to join.

On the VCF Management tab, select the Automation and Operations appliances and click Next.

You can choose identity broker flt-idb01.vcf.sddc.lab to cover both components and click Next.

We tick the acknowledgement and click JOIN.

Let’s confirm both management components are now configured.

To configure Single-Sign on for VCF Operations, we have to navigate to the Other Components tab and click on Add Component.

…

Configure NSX

To give workloads external reachability, we use the guided workflow in vCenter to deploy an NSX Edge cluster hosting a Tier-0 gateway that peers with the physical fabric over BGP, using the Centralized VPC connectivity model.

In the Management Domain vCenter, let’s open Networks > Transit Gateways — the Default Transit Gateway has no connection yet. Start the Setup Network Connectivity workflow.

Here you select Centralized Connection and click Next.

We acknowledge the networking prerequisites and click Next.

We name the Edge cluster m01-ec01, set Tunnel Endpoint MTU 9000 and the Large form factor, then click Add to add the first NSX Edge node.

For node m01-en01, enter the FQDN, cluster m01-cl01. Choose Static IP Assignemtn and set management CIDR 10.230.20.44/24 and gateway; set the Port Group to the VM Management port group.

Scroll down and set the TEP VLAN 60, then click Create New to configure the TEP IP pool for the Edges.

We name the new TEP pool m01-ip-pool02-edge and click Set under Subnets.

Now you click Add Subnet and choose IP Ranges.

Now we enter the TEP range 10.230.60.2–10.230.60.5, CIDR 10.230.60.0/24, gateway 10.230.60.1, then click Add.

You confirm the saved range and click Apply.

With one subnet configured, click Save on the IP pool.

Here you select the new TEP pool for m01-en01 and click Add.

We confirm m01-en01 is listed, then click Add again to add the second NSX Edge node.

For node m01-en02, enter the FQDN, cluster m01-cl01. Choose Static IP Assignemtn and set management CIDR 10.230.20.45/24 and gateway; set the Port Group to the VM Management port group.

We scroll down, set TEP VLAN 60 and select the previously created TEP pool m01-ip-pool02-edge. Then click Apply.

We confirm both edge nodes are listed and click Next.

We define Tier-0 gateway m01-ec01-t0-gw01, HA mode Active/Standby, BGP On, local ASN 65003; then set the first node’s gateway uplinks by clicking on Set.

Configure m01-en01 first uplink – VLAN 70, interface 10.230.70.2/24, BGP peer 10.230.70.1, peer ASN 65001; click Next.

Configure m01-en01 second uplink – VLAN 80, interface 10.230.80.2/24, BGP peer 10.230.80.1; click Apply.

We confirm m01-en01 has two uplinks, then configure m01-en02 by clicking on Set next to m01-en02.

Configure m01-en02 first uplink – VLAN 70, interface 10.230.70.3/24, BGP peer 10.230.70.1; click Save.

Configure m01-en02 second uplink – VLAN 80, interface 10.230.80.3/24, BGP peer 10.230.80.1; click Save.

We confirm both nodes now show two uplinks each and click Next.

In VPC Network Configuration, click Create New for the external IP block.

We create m01-vpc01-ext-ip-block with External visibility and CIDR 192.168.21.0/24; click Save.

Now you click Create New for the private transit-gateway IP block (remove the default day-2 block).

We create m01-vpc01-priv-ip-block with Private visibility and CIDR 172.16.0.0/16; click Save.

You confirm both IP blocks are assigned and click Next.

We review the summary and finally click Deploy.

After deployment, confirm the Transit Gateway has the External Connection assigned with a Success external connection.

Enable Supervisor

The vSphere Supervisor turns the cluster into an IaaS / Kubernetes control plane. It is a requirement for VCF Automation all apps organizations. First create the prerequisites (subscribed content library, vSphere Zone, tag-based storage policy for our NFS storage), then run the Supervisor activation wizard using VCF Networking with VPC.

Let’s open Content Libraries and click CREATE.

You name it “Supervisor Releases Content Library” on m01-vc01 and click Next.

We choose Subscribed content library, paste the depot content-gateway subscription URL of our VCF depot, e.g. https://flt-fc01.vcf.sddc.lab/depot-service/content-gateway/PROD/COMP/SUPERVISOR/lib.json. Then click Next.

You leave the OVF security policy unchecked and click Next.

Here we select the NFS datastore m01-cl01-ds-nfs01 and click Next.

You review the library and click Finish.

Now navigate to Inventory > vCenter level > Configure > vSphere Zones and start creating a zone by clicking Add new vSphere Zone.

We name the zone m01-zn01 and click Next.

You add cluster m01-cl01 to the zone and click Finish.

We confirm zone m01-zn01 is created and mapped to the cluster.

Let’s open Tags & Custom Attributes to create the storage tag.

You create tag “nfs-k8s”, then click New Category.

We create category “storage-k8s” for Datastore objects and click Create.

With nfs-k8s in the storage-k8s category, click Create.

We confirm the tag and category exist.

Open datastore m01-cl01-ds-nfs01 (no tags yet).

We assign the nfs-k8s / storage-k8s tag to the datastore.

You confirm the tag is applied to the datastore.

Let’s open VM Storage Policies and click CREATE.

You name the policy k8s-nfs-default-sp and click Next.

Enable only “Tag based placement rules” and click Next.

You add Rule 1 – use storage tagged nfs-k8s in category storage-k8s; click Next.

We confirm the NFS datastore shows as compatible and click Next.

You review and click Finish.

On the Supervisor Management page, review the prerequisites to begin and click Assign Content Library.

We assign the Supervisor Releases Content Library.

Now start the Supervisor activation by clicking on Get Started.

We begin Supervisor activation — select vCenter m01-vc01 and the VCF Networking with VPC stack; click Next.

You enter Supervisor name m01-sn01, select the zone m01-zn01 and toggle control-plane HA to on; click Next.

We assign the k8s-nfs-default-sp for Control Plane, Ephemeral and Image-Cache storage; click Next.

You set the management network on the vDS mgmt port group — IP range 10.230.20.61-10.230.20.65, mask, gateway, DNS, NTP; click Next.

We set the workload network — default NSX project/VPC, private CIDR 172.25.0.0/16, service CIDR 172.26.0.0/16; click Next.

We can choose the Small control-plane size and API server DNS m01-supervisor.vcf.sddc.lab; click Next.

We review all selections and click Finish.

You watch the Supervisor enter the Configuring state across the four hosts.

Now we wait until the Supervisor shows Running.

Configure VCF Automation

Finally, we prepare VCF Automation for self-service through Provider Management: create a region, an external IP block, a centralized external connection, and then a tenant organization. The Get Started checklist walks us through it.

At the “Welcome to VCF Automation” page, choose Manual Setup.

On Infrastructure Overview, start the Get Started checklist with Create Region.

We create region “region-a” bound to NSX Local Manager m01-nsx01, Supervisor m01-sn01 and storage class k8s-nfs-default-sp; click Create.

We confirm region-a shows Ready.

Now click Start for Create External IP Block.

We enter a name for the external IP block; click Next.

Name it region-a-eipb01.

Add CIDR 192.168.22.0/24; click Next.

We set the default IP quota (e.g. 14 single IPs, 4 CIDRs, /28 max); click Next.

Review and click Create.

You confirm region-a-eipb01 shows Normal.

At 40%, click Start for Create External Connection.

Name the connection region-a-extcon01; click Next.

Select Tier-0 gateway m01-ec01-t0-gw01; click Next.

We associate external IP block region-a-eipb01 and enable Insert Provider Outbound NAT; click Next.

Now choose region-a-eipb01 as the outbound NAT source pool; click Next.

Leave remote networks at 0.0.0.0/0; click Next.

Review and click Create.

We confirm region-a-extcon01 shows Normal (Tier-0, outbound NAT).

Now let’s open Edge and VNA Clusters (none listed yet) and click Sync to sync the Edge cluster we’ve created before.

We confirm edge cluster m01-ec01 is Healthy with 2 nodes.

At 60%, click Start to Create Organization.

We name the tenant “acme”; click Next.

Assign a region quota for region-a via Supervisor m01-sn01; click Add Capacity.

You add zone m01-zn01 capacity (20 GHz CPU limit, 50 GB memory limit); click Add.

We confirm the zone quota, then click Next.

Now select the best-effort VM classes and storage class k8s-nfs-default-sp (100 GB); click Next.

Choose centralized connection region-a-extcon01 for networking; click Next.

Finally add a local user called acme-admin (Organization Administrator); click Create.

We confirm organization “acme” is Enabled and Ready.

As a last step of the VCF Automation Provider setup confirm the Get Started checklist is at 100%.

Last but not least, we log in to the acme tenant portal as acme-admin — we’re now ready to create a namespace and provision workloads. But this is another story 😉

Wrapping Up

At this point the lab is a complete VCF 9.1 fleet: a management domain on nested ESXi, the full VCF Operations suite (logs, network operations, real-time metrics), Single Sign-On, NSX north-south connectivity over BGP, a vSphere Supervisor delivering Kubernetes, and VCF Automation ready for tenant self-service.

That’s all for now 🙂