Updating OS images in a cloud infrastructure is often a tedious and manual task. HashiCorp Packer makes this easier and leveraging CI/CD tools help us to build a strong centralized image pipeline.

In this blog post, I’ll demonstrate how to easily configure Packer to build OS images for Ubuntu Linux and Microsoft Windows Server and how to save these images in an image repository. To automate this build and deploy process, the Packer configuration will be stored in a SCM system, and builds will be automatically triggered using a CI pipeline.

HashiCorp Packer is a tool for creating identical machine images for multiple platforms from a single source configuration. A machine image is a single static unit that contains a pre-configured operating system and installed software which is used to quickly create new running machines. Machine image formats change for each platform. In this demo, we’ll concentrate on the creation and deployment of VMware images, i.e. VMware VM templates (VMDK/VMX files) and OVF templates. 

Apart from the Packer component, we’re using a CI/CD system to further automate the image build and deploy process. In this lab environment, we use Gitlab CE.

The figure below illustrates the overall architecture:

We develop the Packer configuration on our local devbox client and commit changes into the Gitlab repository. We test the configuration by manually run a Gitlab pipeline. Once a given build configuration is considered stable, we create a new tag in Gitlab, which will trigger the pipeline to run automatically. The pipeline checks out the latest Packer build configuration from SCM, and executes the Packer build process.

The general Packer workflow comprises of three high-level steps:

  1. Build: responsible for creating machines on the target cloud platform.
  2. Provision: Once a machine is up and running, provisioners install and configure software within the machine.
  3. Post processing: optional and used to process images after they are built.

In our lab, the build process is comprised of the following steps:

  1. Declare all the required VM configurations in one or more HCL (Hashicorp configuration language) files. This is our Packer template.
  2. Execute Packer with the Packer template as input.
  3. Packer authenticates the vCenter Server and launches a virtual machine based on the Packer template specification.
  4. Packer creates a remote connection to the server (SSH for Linux or WinRM for Windows).
  5. Then it configures the server based on the provisioner we’ve specified in the Packer template (e.g. Shell script, Ansible).
  6. Once provisioning of the VM has been finished, the VM is powered of and Packer creates a template of it, and imports it as a OVF template into the vSphere Content Library.
  7. The virtual machine is deleted in vCenter Server.

Packer setup and configuration

First, we install Packer on our Gitlab Runner, which is in my lab setup a Shell runner on a Ubuntu virtual machine:

$ wget  https://releases.hashicorp.com/packer/1.11.2/packer_1.11.2_linux_amd64.zip
$ unzip packer_1.11.2_linux_amd64.zip
$ sudo mv packer /usr/local/bin/
$ packer version
Packer v1.11.2

For the ISO creation, we install the necessary too mkisofs:

$ sudo apt-get install mkisofs

Next, we setup the Packer template. The Packer template created for this lab has the following structure (you can grab everything from my Github Packer repository, it has been heavily inspired by the fantastic VMware’s Packer examples for vSphere repository):

.
├── manifests
├── README.md
├── scripts
│   ├── linux
│   │   ├── common
│   │   │   └── machine-id.sh
│   │   └── ubuntu
│   │       └── cloud-init.sh
│   └── windows
│       └── common
│           ├── admin-user.ps1
│           ├── cd_files
│           │   ├── vmtools.ps1
│           │   └── winrm.ps1
│           ├── cloudbase-init.ps1
│           └── remote-desktop.ps1
└── vsphere
    ├── build.pkrvars.hcl
    ├── common.pkrvars.hcl
    ├── linux
    │   └── ubuntu
    │       └── 22-04-lts
    │           ├── data
    │           │   ├── meta-data
    │           │   ├── network.pkrtpl.hcl
    │           │   └── user-data.pkrtpl.hcl
    │           ├── linux-ubuntu.auto.pkrvars.hcl
    │           ├── linux-ubuntu.pkr.hcl
    │           └── variables.pkr.hcl
    ├── network.pkrvars.hcl
    ├── vsphere.pkrvars.hcl
    └── windows
        └── server
            └── 2022
                ├── data
                │   ├── autounattend.pkrtpl.hcl
                │   └── network.pkrtpl.hcl
                ├── variables.pkr.hcl
                ├── windows-server.auto.pkrvars.hcl
                └── windows-server.pkr.hcl

The general configuration files for a build reside directly in the vsphere folder:

  • build.pkrvars.hcl: Build account variables.
  • common.pkrvars.hcl: Common variables used for all builds.
  • network.pkrvars.hcl: Network variables.
  • vsphere.pkrvars.hcl: VMware vSphere specific variables.

The different OS types have their own folders, i.e. a linux folder and a windows folder. Inside these folders the different flavors again have their own folders, e.g. Windows Server 2022 in windows/server/2022:

  • The data folder: Contains files to enable an automated installation of the OS using the respective OS installer, e.g. user-data for Ubuntu’s Subiquity, or the answer file for the Windows installer.
  • The pkrvars.hcl file contains OS-specific values for certain build options, e.g. the ISO image to use.
  • The variables.pkr.hcl file contains definitions for all of the variables used.
  • The os.pkr.hcl file is the main template file (e.g. windows-server.pkr.hcl). It contains all the build definitions.

Outside the vsphere folder, inside the scripts folder, we find OS-specific scripts which are either triggered during the unattended installation from within the provisioned machine, or they are executed by the Packer provisioner using SSH or WinRM.

Gitlab pipeline setup

To automate the building of the OS images, we use a Gitlab pipeline, which runs by creating a tag or by scheduling the pipeline. It is located in the root folder of our project and named .gitlab-ci.yml:

workflow:
  rules:
    - if: $CI_COMMIT_TAG
    - if: $CI_PIPELINE_SOURCE == 'schedule'

variables:
  BASE_PATH: "${CI_PROJECT_DIR}/vsphere"
  INPUT_PATH: ""
  BUILD_ONLY: ""

stages:
  - prereq
  - deploy

binaries:
  stage: prereq
  script: 
    - packer version
    - mkisofs -version

.packer-build:
  stage: deploy
  before_script:
    - packer init ${INPUT_PATH}
    - |
      packer validate \
      -var-file="vsphere/build.pkrvars.hcl" \
      -var-file="vsphere/common.pkrvars.hcl" \
      -var-file="vsphere/network.pkrvars.hcl" \
      -var-file="vsphere/vsphere.pkrvars.hcl" \
      ${INPUT_PATH}
  script:
    - |
      packer build -force -on-error=ask \
      -var-file="vsphere/build.pkrvars.hcl" \
      -var-file="vsphere/common.pkrvars.hcl" \
      -var-file="vsphere/network.pkrvars.hcl" \
      -var-file="vsphere/vsphere.pkrvars.hcl" \
      ${BUILD_ONLY} ${INPUT_PATH}
  artifacts:
    paths:
      - manifests/*.json

ubuntu-server-22-04-lts:
  extends: .packer-build
  variables:
    INPUT_PATH: "${BASE_PATH}/linux/ubuntu/22-04-lts/"

windows-server-2022-standard:
  extends: .packer-build
  variables:
    INPUT_PATH: "${BASE_PATH}/windows/server/2022/"
#    BUILD_ONLY: "--only vsphere-iso.windows-server-standard-dexp,vsphere-iso.windows-server-standard-core"
    BUILD_ONLY: "--only vsphere-iso.windows-server-standard-dexp"

Demo time

We trigger the Packer build process by creating a new tag in our Gitlab repository:

A Pipeline is triggered and the two jobs for our Ubuntu and Windows OS can be executed:

Here’s how the successful run of the build process of an Ubuntu template looks like:

If we check the vSphere Content Library, we can see that the two OVF templates have been successfully imported and are ready to be consumed.