We can install a guest operating system in a VM and control guest operating system customisation for VMs by either running commands directly or, if deploying to vSphere-based cloud platforms, through customisation specifications.
In Aria Automation these commands are defined in a so-called cloudConfig resource property in the cloud template code which holds the commands that should be run. For customisation specifications on the other hand, a property in the cloud template code references a vSphere customisation specification by name.
In this post I’m going to show how to leverage Aria Automation cloudConfig to customise Ubuntu 22 and Windows Server 2022 VMs. Such customisation could include:
- Set the hostname
- Set the timezone
- Configure the network adapter
- Partition, format, mount hard disks
- Run arbitrary commands (e.g. join a domain)
Aria Automation cloudConfig in a nutshell
The above mentioned guest customisation method was not designed to provide the full breadth of initialisation customisation that may be required. It is primarily used to configure networking aspects of a VM.
In order to fully customise the VM, we can leverage the Automation Assembler cloudConfig resource property in conjunction with Cloud-init.
Cloud-init is the de-facto industry standard method for cross-platform cloud instance initialisation. It is supported by all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installation. It enables the automated provisioning of an operating system image into a fully running state, complete with required access and applications. Typical tasks handled by cloud-init include networking and storage initialisation, optional package installation and configuration, user account creation and security key provisioning for remote access.
Cloud-init is a daemon which consists of a set of Python scripts and tools to initialize cloud instances of Linux machines.
For Windows there exists an equivalent with Cloudbase-init. It has a different syntax but does essentially the same as Cloud-init.
CloudConfig is VMware’s implementation of Cloud-init. The Automation Assembler cloudConfig resource property uses the declarative YAML syntax to instruct Cloud-init with the necessary tasks to customise the guest OS of the deployed instance.
With Automation Assembler cloudConfig the VM guest OS initialisation is roughly composed of the following steps:
- The Automation Assembler cloudConfig property is picked up by the Aria Automation Provisioning & Lifecycle Service, which performs some pre-processing, e.g. expands input properties, resource properties, and property groups when defined.
- The user data is sent to the so-called Photon endpoint adapters of the Provisioning Service.
- These adapters (e.g vSphere or AWS) use the corresponding APIs to initiate the deployment of a VM from the OS disk image with the provided user data (Cloud-init/Cloudbase-init must be pre-installed in the image template and set to start automatically during the boot process).
- Automation Assembler can leverage the so-called ISO transport method of the OVF datasource to push the Cloud-init user data to the Cloud-init service by mounting an ISO image wich contains this user data in an XML formatted file. The user data itself is stored as a Base64 encoded string in this file, as well as in the VM’s vApp properties.
Note, with the VMware Guestinfo datasource there exists also another possibility to push user data to Cloud-init, but this is out of scope for this blog post. - The VM boots and uses the datasource to access the user data.
When deploying to vSphere, it is recommended to not combine embedded cloudConfig commands and customisation specification initialisation. They aren’t formally compatible and might produce inconsistent or unwanted results when used together.
Ubuntu Linux deployment and customisation on vSphere
In the first part of this session, we’re going to have a closer look an customising an Ubuntu Linux deployment.
Preparing the Ubuntu Linux VM vSphere template
To create a vSphere template that supports Cloud-init, we have to perform the following tasks:
- Deploy a basic Ubuntu 22 VM as the template (VMware tools must be installed)
- Make sure to set the CD-ROM on the VM template to passthrough mode
- Install Cloud-init on it using
sudo apt install cloud-init - Modify /etc/cloud/cloud.cfg as follows
- Disable traditional VMware Linux guest customisation via customisation specification by setting disable_vmware_customization to true
- Define the OVF transport as the only datasource
- Disable Cloud-init network configuration (we’ll do this via Cloud-init’s runcmd module instead)
- End the configuration by executing the cloud-init clean command
- Shut down the VM and convert it into a template
Here is the relevant configuration in /etc/cloud/cloud.cfg:
disable_vmware_customization: true
datasource_list: [ OVF ]
datasource:
OVF:
allow_raw_data: true
vmware_cust_file_max_wait: 25
network:
config: disabledAfter the vSphere template has been created, we have to create the image mapping in Automation Assembler. For this session, we call it “Ubuntu22“.
Creating the Automation Assembler template for Ubuntu
We create a Cloud template which is capable of deploying Ubuntu VMs with the following guest OS customisations:
- Set the hostname
- Configure the network adapter
- Partition, format, mount up to 4 additional hard disks
- Set the root password as specified by the requesting user
The Cloud template looks as follows:
formatVersion: 1
inputs:
osFlavor:
type: string
title: OS flavor
oneOf:
- title: Ubuntu 22.04 LTS
const: Ubuntu22
default: Ubuntu22
vCPU:
type: integer
title: Number of CPUs
enum:
- 2
- 4
- 8
- 16
default: 2
vRAM:
type: integer
title: Memory in MB
enum:
- 1024
- 2048
- 4096
- 8192
- 16384
- 32768
default: 2048
additionalDisks:
type: array
title: Additional Disks
minItems: 0
maxItems: 4
items:
type: object
properties:
diskUnit:
type: integer
title: Disk Number
minimum: 1
maximum: 4
mountPoint:
type: string
title: Mountpoint
minLength: 2
maxLength: 64
pattern: /[a-z0-9A-Z-_]+
diskSize:
type: number
title: Size (GB)
minimum: 1
maximum: 500
rootPassword:
type: string
title: Root Password
minLength: 8
maxLength: 64
encrypted: true
resources:
Cloud_Net_1:
type: Cloud.Network
properties:
networkType: existing
constraints:
- tag: net:dev
Disks:
type: Cloud.vSphere.Disk
allocatePerInstance: true
properties:
capacityGb: ${input.additionalDisks[count.index].diskSize}
SCSIController: SCSI_Controller_0
unitNumber: ${input.additionalDisks[count.index].diskUnit}
count: ${length(input.additionalDisks)}
Cloud_VM_1:
type: Cloud.vSphere.Machine
properties:
image: ${input.osFlavor}
cpuCount: ${input.vCPU}
totalMemoryMB: ${input.vRAM}
folderName: vRA deployed VMs
storage:
constraints:
- tag: storage:silver
networks:
- network: ${resource.Cloud_Net_1.id}
assignment: static
attachedDisks: ${map_to_object(resource.Disks[*].id, "source")}
constraints:
- tag: cz:vsphere
customizeGuestOs: false
cloudConfig: |
#cloud-config
Note, that we’re referencing to a Cloud network “Cloud_Net_1” with the constraint “net:dev“, and to a VM with constraint “cz:vsphere” and storage constraint “storage:silver“.
It is important to set the customizeGuestOs property to false.
Now, on to the cloudConfig property itself.
A pipe character (|) must occur after our cloudConfig directive. All commands after the pipe character are sent to the Cloud-init service in the guest OS after the image is deployed. They are run only on first boot.
The cloudConfig user data section will look as follows:
#cloud-config
write_files:
- path: /etc/netplan/99-installer-config.yaml
content: |
network:
version: 2
renderer: networkd
ethernets:
ens192:
addresses:
- ${self.networks[0].address}/${resource.Cloud_Net_1.prefixLength}
gateway4: ${resource.Cloud_Net_1.gateway}
nameservers:
search: ${resource.Cloud_Net_1.dnsSearchDomains}
addresses: ${resource.Cloud_Net_1.dns}
ssh_pwauth: true
disable_root: false
chpasswd:
list: |
root:${input.rootPassword}
ubuntu:${input.rootPassword}
expire: false
bootcmd:
- fdisk -l /dev/sdb; [ $? -eq 0 ] && printf "o\nn\np\n1\n\n\nw\n" | fdisk /dev/sdb
- fdisk -l /dev/sdc; [ $? -eq 0 ] && printf "o\nn\np\n1\n\n\nw\n" | fdisk /dev/sdc
- fdisk -l /dev/sdd; [ $? -eq 0 ] && printf "o\nn\np\n1\n\n\nw\n" | fdisk /dev/sdd
- fdisk -l /dev/sde; [ $? -eq 0 ] && printf "o\nn\np\n1\n\n\nw\n" | fdisk /dev/sde
fs_setup:
- label: ${to_upper(replace(input.additionalDisks[0].mountPoint,"/",""))}
device: /dev/sdb
partition: 1
filesystem: xfs
overwrite: true
- label: ${to_upper(replace(input.additionalDisks[1].mountPoint,"/",""))}
device: /dev/sdc
partition: 1
filesystem: xfs
overwrite: true
- label: ${to_upper(replace(input.additionalDisks[2].mountPoint,"/",""))}
device: /dev/sdd
partition: 1
filesystem: xfs
overwrite: true
- label: ${to_upper(replace(input.additionalDisks[3].mountPoint,"/",""))}
device: /dev/sde
partition: 1
filesystem: xfs
overwrite: true
runcmd:
- mountpointb="${input.additionalDisks[0].mountPoint}"; if [ $mountpointb != "null" ] && [ -z "$(grep -w $mountpointb /etc/fstab)" ] ; then echo "/dev/sdb1 $mountpointb xfs defaults 0 2" >> /etc/fstab ; fi
- mountpointc="${input.additionalDisks[1].mountPoint}"; if [ $mountpointc != "null" ] && [ -z "$(grep -w $mountpointc /etc/fstab)" ] ; then echo "/dev/sdc1 $mountpointc xfs defaults 0 2" >> /etc/fstab ; fi
- mountpointd="${input.additionalDisks[2].mountPoint}"; if [ $mountpointd != "null" ] && [ -z "$(grep -w $mountpointd /etc/fstab)" ] ; then echo "/dev/sdd1 $mountpointd xfs defaults 0 2" >> /etc/fstab ; fi
- mountpointe="${input.additionalDisks[3].mountPoint}"; if [ $mountpointe != "null" ] && [ -z "$(grep -w $mountpointe /etc/fstab)" ] ; then echo "/dev/sde1 $mountpointe xfs defaults 0 2" >> /etc/fstab ; fi
- mountpointb="${input.additionalDisks[0].mountPoint}"; if [ $mountpointb != "null" ] && [ ! -d $mountpointb ] ; then mkdir -p $mountpointb ; fi
- mountpointc="${input.additionalDisks[1].mountPoint}"; if [ $mountpointc != "null" ] && [ ! -d $mountpointc ] ; then mkdir -p $mountpointc ; fi
- mountpointd="${input.additionalDisks[2].mountPoint}"; if [ $mountpointd != "null" ] && [ ! -d $mountpointd ] ; then mkdir -p $mountpointd ; fi
- mountpointe="${input.additionalDisks[3].mountPoint}"; if [ $mountpointe != "null" ] && [ ! -d $mountpointe ] ; then mkdir -p $mountpointe ; fi
- mount -a
- netplan apply
- hostnamectl set-hostname --static ${self.resourceName}
- eject /dev/cdrom
- touch /etc/cloud/cloud-init.disabled
In the cloudConfig section we’re setting the root account password as defined by the user. Using the write_files module, we create a netplan configuration file called 99-installer-config.yaml, which will be used at a later stage. In the bootcmd module, we create the relevant partition tables depending on the user’s input of selected hard disks. The fs_setup module formats these partitions with the XFS filesystem. Afterwards we run commands in the runcmd module. Here we permanently mount the filesystems, configure networking and set the hostname. Finally we create a file called /etc/cloud/cloud-init.disable, which prevents the Cloud-init service from being start after a reboot of the system.
Windows Server deployment and customisation on vSphere
In the second part of this session, we’re going to have a closer look an customizing a Windows Server 2022 deployment.
Preparing the Windows Server VM vSphere template
The Windows VM image preparation process is similar to the Linux preparation process.
To create a vSphere template that supports Cloudbase-init, we have to
- Deploy a basic Windows Server 2022 VM as the template (VMware tools must be installed)
- Make sure to set the CD-ROM on the VM template to passthrough mode
- Download the Cloudbase-init installer from https://cloudbase.it/downloads/CloudbaseInitSetup_Stable_x64.msi
- Execute the Cloudbase-init GUI installer and run through the wizard
- We’re selecting the following configuration options
- Username: Administrator
- Select the checkbox “Use metadata password“
- User’s local groups: Administrators
- Select the checkbox “Run Cloudbase-init service as LocalSystem“
- Do not close the final page of the setup wizard! With the Completed page of the setup wizard still open, use Windows Explorer to navigate to the Cloudbase-Init installation path, and edit the following files in a text editor:
- conf\cloudbase-init-unattend.conf
Here we set in line 19 metadata_services to cloudbaseinit.metadata.services.ovfservice.OvfService which enables the relevant metadata service. - conf\cloudbase-init.conf
Here we set in line 20 metadata_services to cloudbaseinit.metadata.services.ovfservice.OvfService which enables the relevant metadata service.
- conf\cloudbase-init-unattend.conf
- On the Completed page of the setup wizard, select the options to run Sysprep and to shut down after Sysprep, then click Finish
- We’re selecting the following configuration options
- Shut down the VM and convert it into a template
Here is the content of the cloudbase-init-unattend.conf:
[DEFAULT]
username=Administrator
groups=Administrators
inject_user_password=true
config_drive_raw_hhd=true
config_drive_cdrom=true
config_drive_vfat=true
bsdtar_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\bsdtar.exe
mtools_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\
verbose=true
debug=true
logdir=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\
logfile=cloudbase-init-unattend.log
default_log_levels=comtypes=INFO,suds=INFO,iso8601=WARN,requests=WARN
logging_serial_port_settings=
mtu_use_dhcp_config=true
ntp_use_dhcp_config=true
local_scripts_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts\
metadata_services=cloudbaseinit.metadata.services.ovfservice.OvfService
plugins=cloudbaseinit.plugins.common.mtu.MTUPlugin,cloudbaseinit.plugins.common.sethostname.SetHostNamePlugin,cloudbaseinit.plugins.windows.extendvolumes.ExtendVolumesPlugin
allow_reboot=false
stop_service_on_exit=false
check_latest_version=false
Here is the content of the cloudbase-init.conf:
[DEFAULT]
username=Administrator
groups=Administrators
inject_user_password=true
first_logon_behaviour=no
config_drive_raw_hhd=true
config_drive_cdrom=true
config_drive_vfat=true
bsdtar_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\bsdtar.exe
mtools_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\
verbose=true
debug=true
logdir=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\
logfile=cloudbase-init.log
default_log_levels=comtypes=INFO,suds=INFO,iso8601=WARN,requests=WARN
logging_serial_port_settings=
mtu_use_dhcp_config=true
ntp_use_dhcp_config=true
local_scripts_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts\
metadata_services=cloudbaseinit.metadata.services.ovfservice.OvfService
plugins=cloudbaseinit.plugins.common.userdata.UserDataPlugin
After the vSphere template has been created, we have to create the image mapping in Automation Assembler. For this session, we call it “WinSrv2022“.
Creating the Automation Assembler template for Windows Server
We create a Cloud template which is capable of deploying Windows Server 2022 VMs with the following guest OS customisations:
- Set the hostname
- Configure the network adapter
- Partition, format, mount up to 4 additional hard disks
- Set the root password as specified by the requesting user
The main part of the cloud template looks quite similar to our Linux cloud template:
formatVersion: 1
inputs:
osFlavor:
type: string
title: OS flavor
oneOf:
- title: Windows Server 2022
const: WinSrv2022
default: WinSrv2022
vCPU:
type: integer
title: Number of CPUs
enum:
- 2
- 4
- 8
- 16
default: 2
vRAM:
type: integer
title: Memory in MB
enum:
- 1024
- 2048
- 4096
- 8192
- 16384
- 32768
default: 4096
additionalDisks:
type: array
title: Additional Disks
minItems: 0
maxItems: 4
items:
type: object
properties:
diskUnit:
type: integer
title: Disk Number
minimum: 1
maximum: 4
diskSize:
type: number
title: Size (GB)
minimum: 1
maximum: 500
rootPassword:
type: string
title: Administrator Password
minLength: 8
maxLength: 64
encrypted: true
resources:
Cloud_Net_1:
type: Cloud.Network
properties:
networkType: existing
constraints:
- tag: net:dev
Disks:
type: Cloud.vSphere.Disk
allocatePerInstance: true
properties:
capacityGb: ${input.additionalDisks[count.index].diskSize}
SCSIController: SCSI_Controller_0
unitNumber: ${input.additionalDisks[count.index].diskUnit}
count: ${length(input.additionalDisks)}
Cloud_VM_1:
type: Cloud.vSphere.Machine
properties:
image: ${input.osFlavor}
cpuCount: ${input.vCPU}
totalMemoryMB: ${input.vRAM}
folderName: vRA deployed VMs
storage:
constraints:
- tag: storage:silver
networks:
- network: ${resource.Cloud_Net_1.id}
assignment: static
attachedDisks: ${map_to_object(resource.Disks[*].id, "source")}
constraints:
- tag: cz:vsphere
customizeGuestOs: false
remoteAccess:
authentication: usernamePassword
username: Administrator
password: ${input.rootPassword}
cloudConfig: |
Again, we’re referencing to a Cloud network “Cloud_Net_1” with the constraint “net:dev“, and to a VM with constraint “cz:vsphere” and storage constraint “storage:silver“.
It is again important to set the customizeGuestOs property to false.
Now to the cloudConfig property itself.
Again the pipe character (|) must occur after the cloudConfig directive. All commands after the pipe character are sent to the Cloudbase-init service in the guest OS after the image is deployed. They are run only on first boot.
The cloudConfig user data section will look as follows:
cloudConfig: |
Content-Type: multipart/mixed; boundary="==NewPart=="
MIME-Version: 1.0
--==NewPart==
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config"
set_hostname: ${self.resourceName}
--==NewPart==
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="vm-init.ps1"
#ps1_sysnative
$adapter = Get-NetAdapter
$adapter | Remove-NetIpAddress -Confirm:$false
$adapter | Remove-NetRoute -Confirm:$false
$adapter | New-NetIpAddress -IpAddress ${self.networks[0].address} -PrefixLength ${resource.Cloud_Net_1.prefixLength} -DefaultGateway ${resource.Cloud_Net_1.gateway}
$adapter | Set-DnsClientServerAddress -ServerAddresses ${replace(replace(to_string(resource.Cloud_Net_1.dns[0]),"]",")"),"[","(")},${replace(replace(to_string(resource.Cloud_Net_1.dns[1]),"]",")"),"[","(")}
$arrNewDisks = Get-Disk | Where-Object{$_.OperationalStatus -eq "Offline" -and $_.PartitionStyle -eq "RAW"}
Foreach($objDisk In $arrNewDisks){
$thisDiskNo = $objDisk.Number
$objPartition = Get-Disk -Number $thisDiskNo | Initialize-Disk -PartitionStyle "GPT" -PassThru | New-Partition -AssignDriveLetter -UseMaximumSize
Format-Volume -DriveLetter $objPartition.DriveLetter -FileSystem "NTFS" -NewFileSystemLabel "Disk $thisDiskNo" -Confirm:$false
}
$drives = Get-WmiObject Win32_Volume -Filter "DriveType=5"
$drives | ForEach-Object { (New-Object -ComObject Shell.Application).Namespace(17).ParseName($_.Name).InvokeVerb("Eject") } -ErrorAction SilentlyContinue
The cloudConfig uses MIME multi-part user data (see https://cloudbase-init.readthedocs.io/en/latest/userdata.html#multi-part-content).
A multipart/mixed MIME message is composed of a mix of different data types. Each body part is delineated by a boundary. The boundary parameter is a text string used to delineate one part of the message body from another, in our example “--==NewPart==“.
With the MIME multi-part format, the user data content will be handled based on the content type. The content type “text/cloud-config” will run cloud-config, whereas the content type “text/x-shellscript” executes any script, including Batch and PowerShell scripts. In the example cloud template, we’re executing a script called vm-init.ps1, which will contain the code below (until the next boundary). Using the sysnative parameter ps1_sysnative Cloudbase-init will execute the script using the system native Powershell executable.
We’re then setting configuring the network adapter. Afterwards we’re partitioning and formatting the user defined hard disks and assigning a drive letter to them. Finally we’re ejecting the transport ISO image from the CD drive.
Conclusion
Aria Automation Assembler cloudConfig provides a convenient, standardized way to customise the guest OS of VMs.
One thing to note with the used OVF datasource ISO transport is, that although we’re ejecting the CD at the end of our user data, the transport ISO will be automatically connected to the VM by Aria Automation again, every time the VM has been shut down and then started again.
Another thing to consider is the fact, that the cloudConfig user data is saved more or less in clear text in the VM’s vApp properties (as a Base64 representation), which, depending on your security requirements, might make it not suitable to pass sensitive data (like the root password in the example cloud templates in this blog post). Also, Cloud-init and Cloudbase-init save those sensitive data in clear text in their log files (Cloud-init saves this also in its instance data). To work around this, I recommend to not pass sensitive data via cloudConfig, but use the extensibility capabilities of Aria Automation instead (e.g. custom ABX actions or Orchestrator workflows).
You can find the Cloud template YAML files discussed in this post on my Github vRA repository.
Leave a Reply