Enabling vSphere IaaS control plane and TKG clusters in air-gapped environments

Installing vSphere IaaS Control Plane and TKG Clusters in an Air-gapped Environment

Authors: Vic Parker and Navneet Verma

This article provides high-level instructions and best practice recommendations for enabling vSphere IaaS control plane in an air-gapped environment. It additionally provides instructions on creating Kubernetes (TKG) workload clusters and installing Tanzu packages in the workload cluster. The process can be split into four distinct steps, as per the diagram below

Step 1. - (Red) Prepare and download all the relevant binaries to an internet-connected client.

Step 2. - (Purple) This step offers flexibility as it allows you to transfer the files from Step 1 to the air-gapped client. Note: This step may be unique to each air-gapped deployment/restriction, giving you the control to adapt the process to your specific environment.  

Step 3. - (Green) Step up the required infrastructure using the binaries available on the air-gapped client

Step 4. - (Blue) Install the Kubernetes workload cluster (TKG) and deploy the Tanzu packages before onboarding applications. 
 

Step-by-step Guide

Step 1.  File Downloads

1. Prepare the client by downloading/installing the relevant OS packages. We tested this process on a Linux client. Depending on your setup, you may need to adjust the source and relevant binaries. You must also make them available to the air-gapped client/bootstrap machine. In this example, we will also download all the files to the /data folder. You can modify the download destination to fit your requirements. Please note that you will need 50GB+ of disk space to download all the files. Please make sure that the relevant storage is available on the internet-facing client. 

   Command Tool	Installation Method or Location	Version
   IMGPKG	Carvel Tools	1000000
   JQ	Linux Package Manager	v1.26 or greater
   WGET	Linux Package Manager	V1.21.3 or greater
   KUBECTL	Kubernetes Tools	v1.27
   Docker	Linux Package Manager	V24.0.5 or greater
   GOVC	GOVC GitHub Repository	V0.37.2 or greater
   OpenSSL	Linux Package Manager	V3.0.2 or greater
   CURL	Linux Package Manager	V7.81.0 or greater
   Tanzu CLI	Tanzu CLI	V1.1
   Tanzu Plugins	Tanzu Plugin Download-Bundle	V2.2.0

    Table 1: Required tools and OS utilities. 

2. (Optional) vSphere Binaries. Download the relevant vSphere binaries (ESXi/vCenters - if the vSphere infrastructure is not already installed and configured in the air-gapped environment).  It is recommended to upgrade to the latest vCenter version, where possible. Make sure to save the Reference: VMware vSphere 8.0 Release Notes.
3. Networking Binaries. If you have not already decided on the networking design (NSX or VDS-based) and the networking infrastructure is not already present in the air-gapped environment, download the relevant networking binaries. Use the documentation for reference. Refer to the VMware Interoperability Matrix to validate the NSX support matrix and the vSphere with Tanzu Release Notes for the NSX-ALB support matrix. 
4. Kubernetes Images. To create the Kubernetes workload clusters (step 4), you must download a few of the latest supported Photon and Ubuntu Node Images. Refer to the VMware Tanzu Kubernetes releases Release Notes for the list of supported versions on a given vSphere version.  Linked is a sample script that can be executed to download the image files and convert them to a compressed tar archive. This script has been validated on a Linux/MacOS environment. 
5. (Optional) Harbor Container Registry Binary.  If your air-gapped environment does not have a container registry, VMware provides and supports a Harbor container registry. Download the Harbor OVA from the Broadcom support site. 
6. Tanzu Packages. Tanzu Packages lets you use the Tanzu CLI/Carvel Custom Resources to add and manage common services and addons on Kubernetes clusters. They are deployed on the Kubernetes workload clusters (Step 4). At the time of writing this article, the Tanzu Package offers support for these tools and applications -  Please Note: vSphere 8 (v8.0.3, v8.0.2c, v8.0.1) requires Tanzu CLI v1.1.0, the supported version noted in the compatibility guide.  Using a newer release may lead to failed operations.  

Table 2: Tanzu package bundle. 

You must validate the latest tag for the image repository to download the relevant container images.

$ imgpkg tag list -i projects.registry.vmware.com/tkg/packages/standard/repo

When writing this blog, the latest tag is v2.2.0_update.2 or v2024.5.16. Execute the following command to download the entire repo (around 8GB). This will create a tar file that needs to be transferred to the air-gapped site in step 2.

$ ulimit -S -n 4096
$ imgpkg copy -b projects.registry.vmware.com/tkg/packages/standard/repo:v2.2.0_update.2 --to-tar /data/images.tar

7. Tanzu CLI Plugins 

$ tanzu plugin download-bundle --group vmware-vsphere/default --to-tar /data/vsphere-plugin-bundle.tar 

Step 2. Transfer files to air-gapped env.

You will need a bootstrap workstation directly accessing the target air-gapped vSphere environment.  Use your preferred method to copy/move all the files from the download location (/data in this example) to the bootstrap workstation. 

Step 3. Setup the infrastructure

1. (Optional - if not already installed/configured/updated) Using the binaries downloaded in Step 1.2, install/configure/upgrade the vSphere environment. Configure the vSphere cluster with the required HA and DRS settings. 
2. (Optional - if not already installed/configured/updated) Using the binaries downloaded in Step 1.3, install/configure/upgrade your NSX/NSX-ALB networking stack based on your requirements. 
3. Create a local content library on your vCenter.
4. Using the binaries downloaded in Step 1.4, upload the Photon and Ubuntu Node image files to the local content library. Optional: Use the sample commands provided below to automate the upload. It's important to ensure that the naming structure of the TKR images remains consistent when uploading to the target content library. In the example, the name of the local content library is "LocalCL." Change the value as per your requirements. 

$ cd /data
$ tar -xzvf ${tkgrimage}.tar.gz
$ cd ${tkgrimage}
$ govc library.import -n ${tkgrimage} -m=true LocalCL photon-ova.ovf
$ # or
$ govc library.import -n ${tkgrimage} -m=true LocalCL ubuntu-ova.ovf

5. (Optional - If a container registry is unavailable in the air-gapped environment.) Create DNS entry for Harbor instance, including FQDN and IP. Using the Harbor OVA downloaded in Step   deploy and configure the Harbor repository (during OVA configuration, leave the self-signed option box checked)
   1. Optional - You can bring your certificate or generate your own self-signed certificate (sample script provided here).  If so, you must modify the Harbor configuration and restart the service to point to the certificate/key pair. 
   2. Also note that, as you may execute commands on your bootstrap workstation that accesses the private registry, your bootstrap workstation may also trust the CA when using self-signed. This process may vary depending on your Linux distribution of choice.
6. Create storage tags, tag groups, and storage policies
7. Using the binaries downloaded in Step 1.6, upload the Tanzu packages to the private registry. Replace the {{INTERNAL_IMAGE_REGISTRY}} value in the command below with the name of your private registry.  

$ ulimit -S -n 4096

$ # (use the next command only if you have a self signed certicate for your registry)
$ openssl s_client -showcerts -connect {{INTERNAL_IMAGE_REGISTRY}}:443  </dev/null 2>/dev/null|sed -ne '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' > /tmp/ca.cert
$ imgpkg copy -b /data/images.tar \
      --to-repo {{INTERNAL_IMAGE_REGISTRY}}/tkg/packages/standard/repo \
      --registry-username {{USERID}} \
      --registry-password {{PASSWORD}} \
      --registry-ca-cert-path=/tmp/ca.cert

8. Using the binaries downloaded in Step 1.8, upload Tanzu CLI Plugins to the private registry. Replace the {{INTERNAL_IMAGE_REGISTRY}} value in the command below with the name of your private registry. If using Harbor, you must create a library  - tanzu-cli-plugins in this example. Additionally, you update the Tanzu CLI config to download plugins from the new source location. 

$ tanzu config cert add --host {{INTERNAL_IMAGE_REGISTRY}}:443 --ca-certificate /tmp/ca.cert
$ tanzu plugin upload-bundle --tar /data/vsphere-plugin-bundle.tar --to-repo {{INTERNAL_IMAGE_REGISTRY}}/tanzu-cli-plugins/plugins
$ tanzu plugin source update default --uri {{INTERNAL_IMAGE_REGISTRY}}/tanzu-cli-plugins/plugins/plugin-inventory:latest

Step 4. LCM of Workload clusters. 

1. Following the instructions provided, enable the Supervisor.
2. Create and configure the vSphere Namespace.
3. Deploy a Workload Cluster that trusts the certificate of the container registry that will host the Tanzu Packages and other container images. This could be the optional Harbor registry deployed in Step 3.5 in some environments. 
   1. Sample Cluster YAML: Cluster Class YAML
4. Register local repository with target workload cluster.

$ tanzu context create <workload-cluster> --kubeconfig ~/.kube/config  --kubecontext <workload-cluster> --insecure-skip-tls-verify
$ tanzu context use <workload-cluster>
$ tanzu package repository add tanzu-standard --url {{INTERNAL_IMAGE_REGISTRY}}/tkg/packages/standard/repo:v2.2.0_update.2 --namespace tkg-system
$ tanzu package repository list -A 

5. Deploy Tanzu Packages to the Workload Cluster, for example, Certificate Manager and Contour.
   1. Check Tanzu package compatibility via the Tanzu Standard Repository Release Notes.

$ tanzu package install cert-manager --package-name cert-manager.tanzu.vmware.com --version <version> --namespace tkg-system
$ tanzu package install contour      --package-name contour.tanzu.vmware.com      --version <version> --namespace tkg-system