February 15, 2023

vSphere 8: VMService Bring Your Own Image - Part 1

The VMService builds on the Kubernetes support enabled through vSphere with Tanzu to enable DevOps automation of Virtual Machine lifecycle through the Kubernetes API.  That support was limited to VMware curated guest OS images - Photon and Ubuntu.   Now you can build your base image using the tools you are comfortable with - like Packer or Ansible, then customize the deployed OS using new transports vAppConfig or CloudInit

By Michael West,  Technical Product Manager, VMware

The extensibility of the Kubernetes API brings value to virtual machines as well as containers.  We see this in aspects of lifecycle management. For example, you may find the LCM of an orchestration platform like Kubernetes lines up with your DevOps automation processes, and want to unify how this happens across IT using an API.   Central to this automation is the ability to use the OS distribution of choice and engage your preferred tooling to deliver configured instances on demand.  The VMService is central to DevOps/Developer workflows - providing self-service workload management without VI admin intervention.  In vSphere 7  support was limited to VMware curated guest OS images - Photon and Ubuntu.   Now you can build your base images using the tools you are comfortable with - like Packer or Ansible, then customize the deployed OS using the CloudInit transport.  This blog is the first in a multipart series on the VMware VMService that describes the new capabilities available in vSphere 8 and additional capability coming shortly.  In part 1 of this blog series we will see what was available to customize your VM instances in vSphere 7 and set the stage for part 2 to describe enhancements that were made available in vSphere 8.   For those of you that wish to jump straight to a video demonstration, here you go.

Most of you are probably familiar with deploying a VM through OVF.  You use either the vSphere Client or Ovftool to provide networking and other basic VM configuration as defined in the OVF file.  That information is visible in the vSphere Client under the VM's Configure -> vApp Options -> Properties.

image-20230127074908-1

This method can also be used with VMs deployed through the VMService.  Your VM Specification defined in a YAML file will include the transport method to be used.  Notice OvfEnv is the transport in this VM Specification.  With this approach the VM image must contain VMTools and use the VMware Guest Customization API to pull configuration information into the OS and apply it.  Guest Customization was not designed to provide the full breadth of initialization customization that may be required.  It is primarily used to configure the VM's networking.   

apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachine
metadata:
  labels:
    vm-selector: vmware-tanzu-jumpbox
  name: vmware-tanzu-jumpbox
  namespace: tkg
spec:
  imageName: ubuntu-20-1633387172196
  className: best-effort-xsmall
  powerState: poweredOn
  storageClass: k8s-shared
  advancedOptions:
    defaultVolumeProvisioningOptions:
      thinProvisioned: true
  volumes:
  - name: jumpbox-vol
    persistentVolumeClaim:
      claimName: jumpbox-pvc
  networkInterfaces:
  - networkType: vsphere-distributed
    networkName: k8s-workload
  readinessProbe:
    tcpSocket:
      port: 22
  vmMetadata:
    configMapName: jumpbox-configmap
    transport: OvfEnv

In order to fully customize the VM, users can take advantage of technologies like CloudInit.   CloudInit is an industry standard for instance initialization across OS distributions and clouds.  Prior to vSphere 8, users wishing to fully initialize their VMs through the VMService would use the OvfEnv Transport, but also pass CloudInit configuration into the VM to handle everything beyond the basic properties like Networking. 

The first step in this method is to define the Cloud-Config that will be passed into the VM.  This isn't meant to be an exhaustive tutorial on Cloud-Config, but a quick sample of what you might do.  In this case we are configuring a jumpbox VM that will be used as a Kubernetes client environment.  That environment will be accessible from your web browser through use of the TTYD Service.  Cloud-Config is organized into Modules.  Notice in the User module we have defined a user VMware and assigned its password.   The Write-Files module is where we create the systemd configuration for the ttyd.service.   After updating our package repo sources in the APT module, we define the packages we want installed in PACKAGES.  

 

#cloud-config

package_update: true
package_upgrade: true

ssh:
  emit_keys_to_console: false

no_ssh_fingerprints: true

groups:
  - docker

system_info:
  default_user:
    groups: [docker]

ssh_pwauth: true
users:
  - name: vmware
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    # Password set to Admin!23
    passwd: '$1$salt$SOC33fVbA/ZxeIwD5yw1u1'
    shell: /bin/bash

write_files:
  - path: /etc/systemd/system/ttyd.service
    content: |
      [Unit]
      Description=TTYD
      After=syslog.target
      After=network.target
      [Service]
      ExecStart=/usr/local/bin/ttyd --ssl --ssl-cert /etc/ttyd/certs/server.crt --ssl-key /etc/ttyd/certs/server.key --ssl-ca /etc/ttyd/certs/ca.crt login
      Type=simple
      Restart=always
      User=root
      Group=root
      [Install]
      WantedBy=multi-user.target
    owner: 'root:root'
    permissions: '0644'

apt:
  sources:
    docker.list:
      source: deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
    kubernetes.list:
      source: deb [arch=amd64] https://apt.kubernetes.io/ kubernetes-xenial main
      keyid: 59FE0256827269DC81578F928B57C5C2836F4BEB
    helm-stable-debian.list:
      source: deb [arch=amd64] https://baltocdn.com/helm/stable/debian/ all main
      keyid: 81BF832E2F19CD2AA0471959294AC4827C1A168A

packages:
  - apt-transport-https
  - ca-certificates
  - curl
  - gnupg
  - lsb-release
  - wget
  - docker-ce
  - docker-ce-cli
  - containerd.io
  - net-tools
  - kubectl
  - inetutils-traceroute
  - zip
  - jq
  - zsh
  - bash-completion
  - helm

disk_setup:
  /dev/sdb:
    table_type: gpt
    layout: True
    overwrite: True

fs_setup:
  - device: /dev/sdb
    filesystem: ext4
    partition: 1

mounts:
  - [ /dev/sdb1, /data, "auto", "defaults,noexec,nofail" ]

runcmd:
  - curl -ks https://192.168.220.67/wcp/plugin/linux-amd64/vsphere-plugin.zip -o /tmp/vsphere-plugin.zip
  - unzip -qn /tmp/vsphere-plugin.zip -d /tmp
  - install /tmp/bin/kubectl-vsphere /usr/local/bin
  - wget https://github.com/tsl0922/ttyd/releases/download/1.6.3/ttyd.x86_64 -O /tmp/ttyd
  - install /tmp/ttyd /usr/local/bin
  - systemctl start ttyd && sudo systemctl enable ttyd

The next step is to define the disk and filesystem setup.  In our VM YAML you will see that we have a Persistent Volume that will be attached to the VM and then mounted on /data in the MOUNTS section here.  Finally, we pull and install kubectl from the deployed Supervisor cluster and install TTYD.

Now that we have defined the Cloud-Init configuration we need, let's see how we pass it to the VM.   We have to start by base64 encoding the Cloud-Config so we can include it in a Kubernetes configmap using a command similar to this.  

ubuntu@cli-vm:~/demo-applications$ base64 -w 0 vm-cloud-config.txt

I2Nsb3VkLWNvbmZpZwoKcGFja2FnZV91cGRhdGU6IHRydWUKcGFja2FnZV91cGdyYWRlOiB0cnVlCgpzc2g6CiAgZW1pdF9rZXlzX3RvX2NvbnNvbGU6IGZhbHNlCgpub19zc2hfZmluZ2VycHJpbnRzOiB0cnVlCgpncm91cHM6CiAgLSBkb2NrZXIKCnN5c3RlbV9pbmZvOgogIGRlZmF1bHRfdXNlcjoKICAgIGdyb3VwczogW2RvY2tlcl0KCnNzaF9wd2F1dGg6IHRydWUKdXNlcnM6CiAgLSBuYW1lOiB2bXdhcmUKICAgIHN1ZG86IEFMTD0oQUxMKSBOT1BBU1NXRDpBTEwKICAgIGxvY2tfcGFzc3dkOiBmYWxzZQogICAgIyBQYXNzd29yZCBzZXQgdG8gQWRtaW4hMjMKICAgIHBhc3N3ZDogJyQxJHNhbHQkU09DMzNmVmJBL1p4ZUl3RDV5dzF1MScKICAgIHNoZWxsOiAvYmluL2Jhc2gKCndyaXRlX2ZpbGVzOgogIC0gcGF0aDogL2V0Yy9zeXN0ZW1kL3N5c3RlbS90dHlkLnNlcnZpY2UKICAgIGNvbnRlbnQ6IHwKICAgICAgW1VuaXRdCiAgICAgIERlc2NyaXB0aW9uPVRUWUQKICAgICAgQWZ0ZXI9c3lzbG9nLnRhcmdldAogICAgICBBZnRlcj1uZXR3b3JrLnRhcmdldAogICAgICBbU2VydmljZV0KICAgICAgRXhlY1N0YXJ0PS91c3IvbG9jYWwvYmluL3R0eWQgLS1zc2wgLS1zc2wtY2VydCAvZXRjL3R0eWQvY2VydHMvc2VydmVyLmNydCAtLXNzbC1rZXkgL2V0Yy90dHlkL2NlcnRzL3NlcnZlci5rZXkgLS1zc2wtY2EgL2V0Yy90dHlkL2NlcnRzL2NhLmNydCBsb2dpbgogICAgICBUeXBlPXNpbXBsZQogICAgICBSZXN0YXJ0PWFsd2F5cwogICAgICBVc2VyPXJvb3QKICAgICAgR3JvdXA9cm9vdAogICAgICBbSW5zdGFsbF0KICAgICAgV2FudGVkQnk9bXVsdGktdXNlci50YXJnZXQKICAgIG93bmVyOiAncm9vdDpyb290JwogICAgcGVybWlzc2lvbnM6ICcwNjQ0JyAgICAKICAtIHBhdGg6IC90bXAvc2FuLnR4dAogICAgY29udGVudDogfAogICAgICBzdWJqZWN0QWx0TmFtZT1ETlM6bG9jYWxob3N0CiAgICBvd25lcjogJ3Jvb3Q6cm9vdCcKICAgIHBlcm1pc3Npb25zOiAnMDY0NCcgICAgCgphcHQ6CiAgc291cmNlczoKICAgIGRvY2tlci5saXN0OgogICAgICBzb3VyY2U6IGRlYiBbYXJjaD1hbWQ2NF0gaHR0cHM6Ly9kb3dubG9hZC5kb2NrZXIuY29tL2xpbnV4L3VidW50dSBmb2NhbCBzdGFibGUKICAgICAga2V5aWQ6IDlEQzg1ODIyOUZDN0REMzg4NTRBRTJEODhEODE4MDNDMEVCRkNEODgKICAgIGt1YmVybmV0ZXMubGlzdDoKICAgICAgc291cmNlOiBkZWIgW2FyY2g9YW1kNjRdIGh0dHBzOi8vYXB0Lmt1YmVybmV0ZXMuaW8vIGt1YmVybmV0ZXMteGVuaWFsIG1haW4KICAgICAga2V5aWQ6IDU5RkUwMjU2ODI3MjY5REM4MTU3OEY5MjhCNTdDNUMyODM2RjRCRUIKICAgIGhlbG0tc3RhYmxlLWRlYmlhbi5saXN0OgogICAgICBzb3VyY2U6IGRlYiBbYXJjaD1hbWQ2NF0gaHR0cHM6Ly9iYWx0b2Nkbi5jb20vaGVsbS9zdGFibGUvZGViaWFuLyBhbGwgbWFpbgogICAgICBrZXlpZDogODFCRjgzMkUyRjE5Q0QyQUEwNDcxOTU5Mjk0QUM0ODI3QzFBMTY4QQoKcGFja2FnZXM6CiAgLSBhcHQtdHJhbnNwb3J0LWh0dHBzCiAgLSBjYS1jZXJ0aWZpY2F0ZXMKICAtIGN1cmwKICAtIGdudXBnCiAgLSBsc2ItcmVsZWFzZQogIC0gd2dldAogIC0gZG9ja2VyLWNlCiAgLSBkb2NrZXItY2UtY2xpCiAgLSBjb250YWluZXJkLmlvCiAgLSBuZXQtdG9vbHMKICAtIGt1YmVjdGwKICAtIGluZXR1dGlscy10cmFjZXJvdXRlCiAgLSB6aXAKICAtIGpxCiAgLSB6c2gKICAtIGJhc2gtY29tcGxldGlvbgogIC0gaGVsbQoKZGlza19zZXR1cDoKICAvZGV2L3NkYjoKICAgIHRhYmxlX3R5cGU6IGdwdAogICAgbGF5b3V0OiBUcnVlCiAgICBvdmVyd3JpdGU6IFRydWUKCmZzX3NldHVwOgogIC0gZGV2aWNlOiAvZGV2L3NkYgogICAgZmlsZXN5c3RlbTogZXh0NAogICAgcGFydGl0aW9uOiAxCgptb3VudHM6CiAgLSBbIC9kZXYvc2RiMSwgL2RhdGEsICJhdXRvIiwgImRlZmF1bHRzLG5vZXhlYyxub2ZhaWwiIF0KCnJ1bmNtZDoKICAtIGN1cmwgLWtzIGh0dHBzOi8vMTkyLjE2OC4yMjAuNjcvd2NwL3BsdWdpbi9saW51eC1hbWQ2NC92c3BoZXJlLXBsdWdpbi56aXAgLW8gL3RtcC92c3BoZXJlLXBsdWdpbi56aXAKICAtIHVuemlwIC1xbiAvdG1wL3ZzcGhlcmUtcGx1Z2luLnppcCAtZCAvdG1wCiAgLSBpbnN0YWxsIC90bXAvYmluL2t1YmVjdGwtdnNwaGVyZSAvdXNyL2xvY2FsL2JpbgogIC0gcnVudXNlciAtbCB1YnVudHUgLWMgJ3NoIC1jICIkKGN1cmwgLWZzU0wgaHR0cHM6Ly9yYXcuZ2l0aHViLmNvbS9vaG15enNoL29obXl6c2gvbWFzdGVyL3Rvb2xzL2luc3RhbGwuc2gpIicKICAtIGNoc2ggLXMgJCh3aGljaCB6c2gpIHVidW50dSAKICAtIHJ1bnVzZXIgLWwgdWJ1bnR1IC1jICdzZWQgLWkgInMvWlNIX1RIRU1FPVwicm9iYnlydXNzZWxsXCIvWlNIX1RIRU1FPVwiYWdub3N0ZXJcIi8iIC56c2hyYycKICAtIHdnZXQgaHR0cHM6Ly9naXRodWIuY29tL3Ztd2FyZS9nb3Ztb21pL3JlbGVhc2VzL2Rvd25sb2FkL3YwLjI3LjIvZ292Y19MaW51eF94ODZfNjQudGFyLmd6IC1PIC90bXAvZ292Y19MaW51eF94ODZfNjQudGFyLmd6CiAgLSB0YXIgLXh6dmYgL3RtcC9nb3ZjX0xpbnV4X3g4Nl82NC50YXIuZ3ogLUMgL3RtcAogIC0gaW5zdGFsbCAvdG1wL2dvdmMgL3Vzci9sb2NhbC9iaW4gCiAgLSB3Z2V0IGh0dHBzOi8vZ2l0aHViLmNvbS92bXdhcmUtdGFuenUvcGlubmlwZWQvcmVsZWFzZXMvZG93bmxvYWQvdjAuMTIuMC9waW5uaXBlZC1jbGktbGludXgtYW1kNjQgLU8gL3RtcC9waW5uaXBlZAogIC0gaW5zdGFsbCAvdG1wL3Bpbm5pcGVkIC91c3IvbG9jYWwvYmluCiAgLSB3Z2V0IC1PLSBodHRwczovL2NhcnZlbC5kZXYvaW5zdGFsbC5zaCB8IGJhc2gKICAtIHdnZXQgaHR0cHM6Ly9naXRodWIuY29tL3Ztd2FyZS10YW56dS92ZWxlcm8vcmVsZWFzZXMvZG93bmxvYWQvdjEuNy4wL3ZlbGVyby12MS43LjAtbGludXgtYW1kNjQudGFyLmd6IC1PIC90bXAvdmVsZXJvLXYxLjcuMC1saW51eC1hbWQ2NC50YXIuZ3oKICAtIHRhciAteHp2ZiAvdG1wL3ZlbGVyby12MS43LjAtbGludXgtYW1kNjQudGFyLmd6IC1DIC90bXAKICAtIGluc3RhbGwgL3RtcC92ZWxlcm8tdjEuNy4wLWxpbnV4LWFtZDY0L3ZlbGVybyAvdXNyL2xvY2FsL2JpbiAKICAtIHdnZXQgaHR0cHM6Ly9naXRodWIuY29tL3RzbDA5MjIvdHR5ZC9yZWxlYXNlcy9kb3dubG9hZC8xLjYuMy90dHlkLng4Nl82NCAtTyAvdG1wL3R0eWQKICAtIGluc3RhbGwgL3RtcC90dHlkIC91c3IvbG9jYWwvYmluCiAgLSBta2RpciAtcCAvZXRjL3R0eWQvY2VydHMKICAtIG9wZW5zc2wgZ2VucnNhIC1vdXQgL2V0Yy90dHlkL2NlcnRzL2NhLmtleSAyMDQ4CiAgLSBvcGVuc3NsIHJlcSAtbmV3IC14NTA5IC1kYXlzIDM2NSAta2V5IC9ldGMvdHR5ZC9jZXJ0cy9jYS5rZXkgLXN1YmogIi9DPVVTL1NUPUNBL0w9U0YvTz1WTXdhcmUvQ049VFRZRCBSb290IENBIiAtb3V0IC9ldGMvdHR5ZC9jZXJ0cy9jYS5jcnQKICAtIG9wZW5zc2wgcmVxIC1uZXdrZXkgcnNhOjIwNDggLW5vZGVzIC1rZXlvdXQgL2V0Yy90dHlkL2NlcnRzL3NlcnZlci5rZXkgLXN1YmogIi9DPVVTL1NUPUNBL0w9U0YvTz1WTXdhcmUvQ049bG9jYWxob3N0IiAtb3V0IC9ldGMvdHR5ZC9jZXJ0cy9zZXJ2ZXIuY3NyCiAgLSBvcGVuc3NsIHg1MDkgLXNoYTI1NiAtcmVxIC1leHRmaWxlIC90bXAvc2FuLnR4dCAtZGF5cyAzNjUgLWluIC9ldGMvdHR5ZC9jZXJ0cy9zZXJ2ZXIuY3NyIC1DQSAvZXRjL3R0eWQvY2VydHMvY2EuY3J0IC1DQWtleSAvZXRjL3R0eWQvY2VydHMvY2Eua2V5IC1DQWNyZWF0ZXNlcmlhbCAtb3V0IC9ldGMvdHR5ZC9jZXJ0cy9zZXJ2ZXIuY3J0CiAgLSBzeXN0ZW1jdGwgc3RhcnQgdHR5ZCAmJiBzdWRvIHN5c3RlbWN0bCBlbmFibGUgdHR5ZAo=

Now we define the YAML for the configmap and paste the encoded Cloud-Config into the user-data field.  You might also add authorized SSH keys in the same way.

apiVersion: v1
kind: ConfigMap
metadata:
    name: jumpbox-configmap
    namespace: tkg
data:
  hostname: vmware-tanzu-jumpbox
  public-keys: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC5KYNeWQgVHrDHaEhBCLF1vIR0OAtUIJwjKYkY4E/5HhEu8fPFvBOIHPFTPrtkX4vzSiMFKE5WheKGQIpW3HHlRbmRPc9oe6nNKlsUfFAaJ7OKF146Gjpb7lWs/C34mjdtxSb1D/YcHSyqK5mxhyHAXPeK8lrxG5MLOJ3X2A3iUvXcBo1NdhRdLRWQmyjs16fnPx6840x9n5NqeiukFYIVhDMFErq42AkeewsWcbZQuwViSLk2cIc09eykAjaXMojCmSbjrj0kC3sbYX+HD2OWbKohTqqO6/UABtjYgTjIS4PqsXWk63dFdcxF6ukuO6ZHaiY7h3xX2rTg9pv1oT8WBR44TYgvyRp0Bhe0u2/n/PUTRfp22cOWTA2wG955g7jOd7RVGhtMHi9gFXeUS2KodO6C4XEXC7Y2qp9p9ARlNvu11QoaDyH3l0h57Me9we+3XQNuteV69TYrJnlgWecMa/x+rcaEkgr7LD61dY9sTuufttLBP2ro4EIWoBY6F1Ozvcp8lcgi/55uUGxwiKDA6gQ+UA/xtrKk60s6MvYMzOxJiUQbWYr3MJ3NSz6PJVXMvlsAac6U+vX4U9eJP6/C1YDyBaiT96cb/B9TkvpLrhPwqMZdYVomVHsdY7YriJB93MRinKaDJor1aIE/HMsMpbgFCNA7mma9x5HS/57Imw== admin@corp.local
  user-data: I2Nsb3VkLWNvbmZpZwoKcGFja2FnZV91cGRhdGU6IHRydWUKcGFja2FnZV91cGdyYWRlOiB0cnVlCgpzc2g6CiAgZW1pdF9rZXlzX3RvX2NvbnNvbGU6IGZhbHNlCgpub19zc2hfZmluZ2VycHJpbnRzOiB0cnVlCgpncm91cHM6CiAgLSBkb2NrZXIKCnN5c3RlbV9pbmZvOgogIGRlZmF1bHRfdXNlcjoKICAgIGdyb3VwczogW2RvY2tlcl0KCndyaXRlX2ZpbGVzOgogIC0gcGF0aDogL2V0Yy9zeXN0ZW1kL3N5c3RlbS90dHlkLnNlcnZpY2UKICAgIGNvbnRlbnQ6IHwKICAgICAgW1VuaXRdCiAgICAgIERlc2NyaXB0aW9uPVRUWUQKICAgICAgQWZ0ZXI9c3lzbG9nLnRhcmdldAogICAgICBBZnRlcj1uZXR3b3JrLnRhcmdldAogICAgICBbU2VydmljZV0KICAgICAgRXhlY1N0YXJ0PS91c3IvbG9jYWwvYmluL3R0eWQgLS1zc2wgLS1zc2wtY2VydCAvZXRjL3R0eWQvY2VydHMvc2VydmVyLmNydCAtLXNzbC1rZXkgL2V0Yy90dHlkL2NlcnRzL3NlcnZlci5rZXkgLS1zc2wtY2EgL2V0Yy90dHlkL2NlcnRzL2NhLmNydCBsb2dpbgogICAgICBUeXBlPXNpbXBsZQogICAgICBSZXN0YXJ0PWFsd2F5cwogICAgICBVc2VyPXJvb3QKICAgICAgR3JvdXA9cm9vdAogICAgICBbSW5zdGFsbF0KICAgICAgV2FudGVkQnk9bXVsdGktdXNlci50YXJnZXQKICAgIG93bmVyOiAncm9vdDpyb290JwogICAgcGVybWlzc2lvbnM6ICcwNjQ0JyAgICAKICAtIHBhdGg6IC90bXAvc2FuLnR4dAogICAgY29udGVudDogfAogICAgICBzdWJqZWN0QWx0TmFtZT1ETlM6bG9jYWxob3N0CiAgICBvd25lcjogJ3Jvb3Q6cm9vdCcKICAgIHBlcm1pc3Npb25zOiAnMDY0NCcgICAgCgphcHQ6CiAgc291cmNlczoKICAgIGRvY2tlci5saXN0OgogICAgICBzb3VyY2U6IGRlYiBbYXJjaD1hbWQ2NF0gaHR0cHM6Ly9kb3dubG9hZC5kb2NrZXIuY29tL2xpbnV4L3VidW50dSBmb2NhbCBzdGFibGUKICAgICAga2V5aWQ6IDlEQzg1ODIyOUZDN0REMzg4NTRBRTJEODhEODE4MDNDMEVCRkNEODgKICAgIGt1YmVybmV0ZXMubGlzdDoKICAgICAgc291cmNlOiBkZWIgW2FyY2g9YW1kNjRdIGh0dHBzOi8vYXB0Lmt1YmVybmV0ZXMuaW8vIGt1YmVybmV0ZXMteGVuaWFsIG1haW4KICAgICAga2V5aWQ6IDU5RkUwMjU2ODI3MjY5REM4MTU3OEY5MjhCNTdDNUMyODM2RjRCRUIKICAgIGhlbG0tc3RhYmxlLWRlYmlhbi5saXN0OgogICAgICBzb3VyY2U6IGRlYiBbYXJjaD1hbWQ2NF0gaHR0cHM6Ly9iYWx0b2Nkbi5jb20vaGVsbS9zdGFibGUvZGViaWFuLyBhbGwgbWFpbgogICAgICBrZXlpZDogODFCRjgzMkUyRjE5Q0QyQUEwNDcxOTU5Mjk0QUM0ODI3QzFBMTY4QQoKcGFja2FnZXM6CiAgLSBhcHQtdHJhbnNwb3J0LWh0dHBzCiAgLSBjYS1jZXJ0aWZpY2F0ZXMKICAtIGN1cmwKICAtIGdudXBnCiAgLSBsc2ItcmVsZWFzZQogIC0gd2dldAogIC0gZG9ja2VyLWNlCiAgLSBkb2NrZXItY2UtY2xpCiAgLSBjb250YWluZXJkLmlvCiAgLSBuZXQtdG9vbHMKICAtIGt1YmVjdGwKICAtIGluZXR1dGlscy10cmFjZXJvdXRlCiAgLSB6aXAKICAtIGpxCiAgLSB6c2gKICAtIGJhc2gtY29tcGxldGlvbgogIC0gaGVsbQoKZGlza19zZXR1cDoKICAvZGV2L3NkYjoKICAgIHRhYmxlX3R5cGU6IGdwdAogICAgbGF5b3V0OiBUcnVlCiAgICBvdmVyd3JpdGU6IFRydWUKCmZzX3NldHVwOgogIC0gZGV2aWNlOiAvZGV2L3NkYgogICAgZmlsZXN5c3RlbTogZXh0NAogICAgcGFydGl0aW9uOiAxCgptb3VudHM6CiAgLSBbIC9kZXYvc2RiMSwgL2RhdGEsICJhdXRvIiwgImRlZmF1bHRzLG5vZXhlYyxub2ZhaWwiIF0KCnJ1bmNtZDoKICAtIGN1cmwgLWtzIGh0dHBzOi8vMTkyLjE2OC4yMjAuNjcvd2NwL3BsdWdpbi9saW51eC1hbWQ2NC92c3BoZXJlLXBsdWdpbi56aXAgLW8gL3RtcC92c3BoZXJlLXBsdWdpbi56aXAKICAtIHVuemlwIC1xbiAvdG1wL3ZzcGhlcmUtcGx1Z2luLnppcCAtZCAvdG1wCiAgLSBpbnN0YWxsIC90bXAvYmluL2t1YmVjdGwtdnNwaGVyZSAvdXNyL2xvY2FsL2JpbgogIC0gcnVudXNlciAtbCB1YnVudHUgLWMgJ3NoIC1jICIkKGN1cmwgLWZzU0wgaHR0cHM6Ly9yYXcuZ2l0aHViLmNvbS9vaG15enNoL29obXl6c2gvbWFzdGVyL3Rvb2xzL2luc3RhbGwuc2gpIicKICAtIGNoc2ggLXMgJCh3aGljaCB6c2gpIHVidW50dSAKICAtIHJ1bnVzZXIgLWwgdWJ1bnR1IC1jICdzZWQgLWkgInMvWlNIX1RIRU1FPVwicm9iYnlydXNzZWxsXCIvWlNIX1RIRU1FPVwiYWdub3N0ZXJcIi8iIC56c2hyYycKICAtIHdnZXQgaHR0cHM6Ly9naXRodWIuY29tL3Ztd2FyZS9nb3Ztb21pL3JlbGVhc2VzL2Rvd25sb2FkL3YwLjI3LjIvZ292Y19MaW51eF94ODZfNjQudGFyLmd6IC1PIC90bXAvZ292Y19MaW51eF94ODZfNjQudGFyLmd6CiAgLSB0YXIgLXh6dmYgL3RtcC9nb3ZjX0xpbnV4X3g4Nl82NC50YXIuZ3ogLUMgL3RtcAogIC0gaW5zdGFsbCAvdG1wL2dvdmMgL3Vzci9sb2NhbC9iaW4gCiAgLSB3Z2V0IGh0dHBzOi8vZ2l0aHViLmNvbS92bXdhcmUtdGFuenUvcGlubmlwZWQvcmVsZWFzZXMvZG93bmxvYWQvdjAuMTIuMC9waW5uaXBlZC1jbGktbGludXgtYW1kNjQgLU8gL3RtcC9waW5uaXBlZAogIC0gaW5zdGFsbCAvdG1wL3Bpbm5pcGVkIC91c3IvbG9jYWwvYmluCiAgLSB3Z2V0IC1PLSBodHRwczovL2NhcnZlbC5kZXYvaW5zdGFsbC5zaCB8IGJhc2gKICAtIHdnZXQgaHR0cHM6Ly9naXRodWIuY29tL3Ztd2FyZS10YW56dS92ZWxlcm8vcmVsZWFzZXMvZG93bmxvYWQvdjEuNy4wL3ZlbGVyby12MS43LjAtbGludXgtYW1kNjQudGFyLmd6IC1PIC90bXAvdmVsZXJvLXYxLjcuMC1saW51eC1hbWQ2NC50YXIuZ3oKICAtIHRhciAteHp2ZiAvdG1wL3ZlbGVyby12MS43LjAtbGludXgtYW1kNjQudGFyLmd6IC1DIC90bXAKICAtIGluc3RhbGwgL3RtcC92ZWxlcm8tdjEuNy4wLWxpbnV4LWFtZDY0L3ZlbGVybyAvdXNyL2xvY2FsL2JpbiAKICAtIHdnZXQgaHR0cHM6Ly9naXRodWIuY29tL3RzbDA5MjIvdHR5ZC9yZWxlYXNlcy9kb3dubG9hZC8xLjYuMy90dHlkLng4Nl82NCAtTyAvdG1wL3R0eWQKICAtIGluc3RhbGwgL3RtcC90dHlkIC91c3IvbG9jYWwvYmluCiAgLSBta2RpciAtcCAvZXRjL3R0eWQvY2VydHMKICAtIG9wZW5zc2wgZ2VucnNhIC1vdXQgL2V0Yy90dHlkL2NlcnRzL2NhLmtleSAyMDQ4CiAgLSBvcGVuc3NsIHJlcSAtbmV3IC14NTA5IC1kYXlzIDM2NSAta2V5IC9ldGMvdHR5ZC9jZXJ0cy9jYS5rZXkgLXN1YmogIi9DPVVTL1NUPUNBL0w9U0YvTz1WTXdhcmUvQ049VFRZRCBSb290IENBIiAtb3V0IC9ldGMvdHR5ZC9jZXJ0cy9jYS5jcnQKICAtIG9wZW5zc2wgcmVxIC1uZXdrZXkgcnNhOjIwNDggLW5vZGVzIC1rZXlvdXQgL2V0Yy90dHlkL2NlcnRzL3NlcnZlci5rZXkgLXN1YmogIi9DPVVTL1NUPUNBL0w9U0YvTz1WTXdhcmUvQ049bG9jYWxob3N0IiAtb3V0IC9ldGMvdHR5ZC9jZXJ0cy9zZXJ2ZXIuY3NyCiAgLSBvcGVuc3NsIHg1MDkgLXNoYTI1NiAtcmVxIC1leHRmaWxlIC90bXAvc2FuLnR4dCAtZGF5cyAzNjUgLWluIC9ldGMvdHR5ZC9jZXJ0cy9zZXJ2ZXIuY3NyIC1DQSAvZXRjL3R0eWQvY2VydHMvY2EuY3J0IC1DQWtleSAvZXRjL3R0eWQvY2VydHMvY2Eua2V5IC1DQWNyZWF0ZXNlcmlhbCAtb3V0IC9ldGMvdHR5ZC9jZXJ0cy9zZXJ2ZXIuY3J0CiAgLSBzeXN0ZW1jdGwgc3RhcnQgdHR5ZCAmJiBzdWRvIHN5c3RlbWN0bCBlbmFibGUgdHR5ZAo=

You can now create the configmap and the VM by the kubectl apply -f "Filename" command.  The user-data will be passed into the VM using the OvfEnv Transport and will  also be visible in the vApp Options of the VM.

Downside of OvfEnv Transport with Cloud-Init

As mentioned earlier, this method combines GOSC (Guest Customization) with Cloud-Init.  When the VM initially boots, both processes attempt to initialize the VM and their ordering can be unpredictable.  Race conditions can cause VM initialization to be non-deterministic and fail.  VM reboots may also cause a loss of configuration.  VMware curated images for PhotonOS and Ubuntu have been patched to eliminate this issue, but non-patched OS Images would not be supported.  That is why prior to vSphere 8, Bring Your Own Image (BYOI) only supported those curated images.  In Part 2 of this blog we will see what has changed in vSphere 8 and how you can bring any Linux image and initialize its configuration through the VMService and Cloud-Init.

For more detail on configuring your VM through the process above, check out this video.  https://youtu.be/7xfMie864_M

Filter Tags

Modern Applications vSphere 8 vSphere with Tanzu Kubernetes VM Service Blog Deep Dive Deployment Considerations Technical Overview Intermediate Advanced

Michael West

Read More from the Author

Michael is a Staff Technical Product Manager in the Cloud Platform Business Unit and has been with VMware for 13 years. He is focused on technologies like Kubernetes that operationalize container based applications at scale. When not ruining his eyesight while staring at small font CLIs, he enjoys fitness training, stand up paddle boarding and his Staffordshire Bull Terrier - Elvis.