November 07, 2022

TKG 2 - What's New with TKRs

By Michael West, VMware vSphere 8 introduces an upgrade to the Tanzu Kubernetes Grid (TKG 2).   One of the areas of change is in the Tanzu Kubernetes Release (TKR).  A TKR is a VM base image - packaged as an OVA file - providing a Kubernetes software distribution signed and supported by VMware for use in Kubernetes clusters provisioned by TKG.  The naming is aligned with Kubernetes upstream versions, so it it easy to deploy clusters with your preferred version of Kubernetes.   As with vSphere 7, the TKRs are available on a public Content Delivery Network (CDN).  You simply create a Content Library in vCenter that is subscribed to that CDN.

vSphere 8 introduces an upgrade to the Tanzu Kubernetes Grid (TKG 2).   One of the areas of change is in the Tanzu Kubernetes Release (TKR).  Both the format and capability of TKRs have changed and it is important to select appropriate images for your environment.  A TKR is a VM base image - packaged as an OVA file - providing a Kubernetes software distribution signed and supported by VMware for use in Kubernetes clusters provisioned by TKG.  The naming is aligned with Kubernetes upstream versions, so it it easy to deploy clusters with your preferred version of Kubernetes.  

As with vSphere 7, the TKRs are available on a public Content Delivery Network (CDN).  You simply create a Content Library in vCenter that is subscribed to that CDN.

852x555image-20221107080112-1image-20221107080112-2

 

The TKRs automatically sync to the Supervisor cluster.  You can see them by executing:  kubectl get tkr

/home/ubuntu: kubectl get tkr
NAME                                  VERSION                             READY   COMPATIBLE   CREATED
v1.16.12---vmware.1-tkg.1.da7afe7     v1.16.12+vmware.1-tkg.1.da7afe7     False   False        4d1h
v1.16.14---vmware.1-tkg.1.ada4837     v1.16.14+vmware.1-tkg.1.ada4837     False   False        4d1h
v1.16.8---vmware.1-tkg.3.60d2ffd      v1.16.8+vmware.1-tkg.3.60d2ffd      False   False        4d1h
v1.17.11---vmware.1-tkg.1.15f1e18     v1.17.11+vmware.1-tkg.1.15f1e18     False   False        4d1h
v1.17.11---vmware.1-tkg.2.ad3d374     v1.17.11+vmware.1-tkg.2.ad3d374     False   False        4d1h
v1.17.13---vmware.1-tkg.2.2c133ed     v1.17.13+vmware.1-tkg.2.2c133ed     False   False        4d1h
v1.17.17---vmware.1-tkg.1.d44d45a     v1.17.17+vmware.1-tkg.1.d44d45a     False   False        4d1h
v1.17.7---vmware.1-tkg.1.154236c      v1.17.7+vmware.1-tkg.1.154236c      False   False        4d1h
v1.17.8---vmware.1-tkg.1.5417466      v1.17.8+vmware.1-tkg.1.5417466      False   False        4d1h
v1.18.10---vmware.1-tkg.1.3a6cd48     v1.18.10+vmware.1-tkg.1.3a6cd48     False   False        4d1h
v1.18.15---vmware.1-tkg.1.600e412     v1.18.15+vmware.1-tkg.1.600e412     False   False        4d1h
v1.18.15---vmware.1-tkg.2.ebf6117     v1.18.15+vmware.1-tkg.2.ebf6117     False   False        4d1h
v1.18.19---vmware.1-tkg.1.17af790     v1.18.19+vmware.1-tkg.1.17af790     False   False        4d1h
v1.18.5---vmware.1-tkg.1.c40d30d      v1.18.5+vmware.1-tkg.1.c40d30d      False   False        4d1h
v1.19.11---vmware.1-tkg.1.9d9b236     v1.19.11+vmware.1-tkg.1.9d9b236     False   False        4d1h
v1.19.14---vmware.1-tkg.1.8753786     v1.19.14+vmware.1-tkg.1.8753786     False   False        4d1h
v1.19.16---vmware.1-tkg.1.df910e2     v1.19.16+vmware.1-tkg.1.df910e2     False   False        4d1h
v1.19.7---vmware.1-tkg.1.fc82c41      v1.19.7+vmware.1-tkg.1.fc82c41      False   False        4d1h
v1.19.7---vmware.1-tkg.2.f52f85a      v1.19.7+vmware.1-tkg.2.f52f85a      False   False        4d1h
v1.20.12---vmware.1-tkg.1.b9a42f3     v1.20.12+vmware.1-tkg.1.b9a42f3     False   False        4d1h
v1.20.2---vmware.1-tkg.1.1d4f79a      v1.20.2+vmware.1-tkg.1.1d4f79a      False   False        4d1h
v1.20.2---vmware.1-tkg.2.3e10706      v1.20.2+vmware.1-tkg.2.3e10706      False   False        4d1h
v1.20.7---vmware.1-tkg.1.7fb9067      v1.20.7+vmware.1-tkg.1.7fb9067      False   False        4d1h
v1.20.8---vmware.1-tkg.2              v1.20.8+vmware.1-tkg.2              False   False        4d1h
v1.20.9---vmware.1-tkg.1.a4cee5b      v1.20.9+vmware.1-tkg.1.a4cee5b      False   False        4d1h
v1.21.2---vmware.1-tkg.1.ee25d55      v1.21.2+vmware.1-tkg.1.ee25d55      True    True         4d1h
v1.21.6---vmware.1-tkg.1              v1.21.6+vmware.1-tkg.1              True    True         4d1h
v1.21.6---vmware.1-tkg.1.b3d708a      v1.21.6+vmware.1-tkg.1.b3d708a      True    True         4d1h
v1.22.9---vmware.1-tkg.1.cc71bc8      v1.22.9+vmware.1-tkg.1.cc71bc8      True    True         4d1h
v1.23.8---vmware.2-tkg.2-zshippable   v1.23.8+vmware.2-tkg.2-zshippable   True    True         4d1h

​There are a few things to note in this output.  The name is consistent with the Kubernetes version that will run in clusters created with this TKR.   The "Compatible" column shows images that can be used with vSphere 8.    If you are upgrading the Supervisor from vSphere 7, you MUST update your existing Tanzu Kubernetes Clusters (TKC) to one of the compatible TKRs prior to upgrading your Supervisor cluster.   If you try to upgrade your Supervisor with Tanzu Kubernetes Clusters (TKCs) running incompatible images, you will see an incompatible infrastructure error.  

Not all "Compatible" images contain the new TKR format and capability.  We have updated several vSphere 7 TKRs to run in vSphere 8 as an upgrade path, however only images whose name ends in "zshippable" contain the new packaging format and functionality.   You can only use "zshippable" images to create clusters on vSphere 8 updated Supervisors.  For complete information on compatibility check out the VMware Tanzu Kubernetes Releases release notes

What's New in vSphere 8 "zshippable" TKRs?

Multi-OS Support in Single TKR

A single TKR can contain all supported operating system images for a particular Kubernetes release through the OSImage format.  Currently we are supporting PhotonOS and Ubuntu based images.  You can see the supported OSs in the TKR by running:  kubectl tkr "tkr-name" -o yaml  

ownerReferences:
  - apiVersion: vmoperator.vmware.com/v1alpha1
    kind: VirtualMachineImage
    name: ob-20611104-ubuntu-2004-amd64-vmi-k8s-v1.23.8---vmware.2-tkg.1-zshippable
    uid: 8eeac7c0-fa87-4d41-b4c2-615484c92b60
  - apiVersion: vmoperator.vmware.com/v1alpha1
    kind: VirtualMachineImage
    name: ob-20611023-photon-3-amd64-vmi-k8s-v1.23.8---vmware.2-tkg.1-zshippable
    uid: 98f6bccb-c6f9-4cf0-8851-6cce8332b560

Under the ownerReferences section, you see the names of the OSImages contained in this TKR.  You can further drill into those OSImages with:  kubectl get osimages

 /home/ubuntu: kubectl get osimages 
NAME                                                                        K8S VERSION         OS NAME   OS VERSION   ARCH    TYPE   COMPATIBLE   CREATED
ob-15957779-photon-3-k8s-v1.16.8---vmware.1-tkg.3.60d2ffd                   v1.16.8+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-16466772-photon-3-k8s-v1.17.7---vmware.1-tkg.1.154236c                   v1.17.7+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-16545581-photon-3-k8s-v1.16.12---vmware.1-tkg.1.da7afe7                  v1.16.12+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-16551547-photon-3-k8s-v1.17.8---vmware.1-tkg.1.5417466                   v1.17.8+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-16897056-photon-3-k8s-v1.16.14---vmware.1-tkg.1.ada4837                  v1.16.14+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-16924026-photon-3-k8s-v1.18.5---vmware.1-tkg.1.c40d30d                   v1.18.5+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-16924027-photon-3-k8s-v1.17.11---vmware.1-tkg.1.15f1e18                  v1.17.11+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-17010758-photon-3-k8s-v1.17.11---vmware.1-tkg.2.ad3d374                  v1.17.11+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-17332787-photon-3-k8s-v1.17.13---vmware.1-tkg.2.2c133ed                  v1.17.13+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-17419070-photon-3-k8s-v1.18.10---vmware.1-tkg.1.3a6cd48                  v1.18.10+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-17654937-photon-3-k8s-v1.18.15---vmware.1-tkg.1.600e412                  v1.18.15+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-17658793-photon-3-k8s-v1.17.17---vmware.1-tkg.1.d44d45a                  v1.17.17+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-17660956-photon-3-k8s-v1.19.7---vmware.1-tkg.1.fc82c41                   v1.19.7+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-17861429-photon-3-k8s-v1.20.2---vmware.1-tkg.1.1d4f79a                   v1.20.2+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-18035533-photon-3-k8s-v1.18.15---vmware.1-tkg.2.ebf6117                  v1.18.15+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-18035534-photon-3-k8s-v1.19.7---vmware.1-tkg.2.f52f85a                   v1.19.7+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-18037317-photon-3-k8s-v1.20.2---vmware.1-tkg.2.3e10706                   v1.20.2+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-18186591-photon-3-k8s-v1.20.7---vmware.1-tkg.1.7fb9067                   v1.20.7+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-18284400-photon-3-k8s-v1.18.19---vmware.1-tkg.1.17af790                  v1.18.19+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-18324108-photon-3-k8s-v1.19.11---vmware.1-tkg.1.9d9b236                  v1.19.11+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-18461281-photon-3-k8s-v1.20.9---vmware.1-tkg.1.a4cee5b                   v1.20.9+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-18532793-photon-3-k8s-v1.19.14---vmware.1-tkg.1.8753786                  v1.19.14+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-18592554-photon-3-k8s-v1.21.2---vmware.1-tkg.1.ee25d55                   v1.21.2+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-18807685-tkgs-ova-ubuntu-2004-v1.20.8---vmware.1-tkg.2                   v1.20.8+vmware.1    ubuntu    20.04        amd64   vmi                 4d3h
ob-18895415-photon-3-k8s-v1.19.16---vmware.1-tkg.1.df910e2                  v1.19.16+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-18900476-photon-3-k8s-v1.21.6---vmware.1-tkg.1.b3d708a                   v1.21.6+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-18903450-photon-3-k8s-v1.20.12---vmware.1-tkg.1.b9a42f3                  v1.20.12+vmware.1   photon    3.0          amd64   vmi                 4d3h
ob-19344082-tkgs-ova-ubuntu-2004-v1.21.6---vmware.1-tkg.1                   v1.21.6+vmware.1    ubuntu    20.04        amd64   vmi                 4d3h
ob-19930037-photon-3-k8s-v1.22.9---vmware.1-tkg.1.cc71bc8                   v1.22.9+vmware.1    photon    3.0          amd64   vmi                 4d3h
ob-20611023-photon-3-amd64-vmi-k8s-v1.23.8---vmware.2-tkg.1-zshippable      v1.23.8+vmware.2    photon    3            amd64   vmi                 4d3h
ob-20611104-ubuntu-2004-amd64-vmi-k8s-v1.23.8---vmware.2-tkg.1-zshippable   v1.23.8+vmware.2    ubuntu    20.04        amd64   vmi                 4d3h

When creating a TKG cluster, selecting the appropriate image in the manifest involves two specifications.  You must add the name of the TKR and you must add an annotation to choose the OSImage you want to use from that TKR.  Note that you might even choose to run different images for control plane nodes and workers, or even break workers into separate nodePools and choose different OSImages for each nodePool.   An example would be when deploying GPU enabled clusters, NVIDIA currently does not support PhotonOS for its GPU Operator, so GPU nodes could run Ubuntu, while the other workers could be PhotonOS.  Notice the annotation "run.tanzu.vmware.com/resolve-os-image.gpuworkers: os-name=ubuntu".  This specifies that the workers in the gpuworkers nodepool will get ubuntu and since there is no annotation for the nogpuworker nodepool, it gets the default PhotonOS.

apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesCluster
metadata:
  name: tkc-2
  namespace: utilities
  annotations:
          run.tanzu.vmware.com/resolve-os-image.gpuworkers: os-name=ubuntu
spec:
  distribution:
    fullVersion: v1.23.8---vmware.2-tkg.2-zshippable
  topology:
    controlPlane:
      replicas: 1
      storageClass: kubernetes-policy
      tkr:
        reference:
          name: v1.23.8---vmware.2-tkg.2-zshippable
      vmClass: best-effort-small
    nodePools:
    - name: gpuworkers
      replicas: 1
      storageClass: kubernetes-policy
      tkr:
        reference:
          name: v1.23.8---vmware.2-tkg.2-zshippable
      vmClass: gpuclass-a100
      volumes:
      - capacity:
          storage: 50Gi
        mountPath: /var/lib/containerd
        name: containerd
      - capacity:
          storage: 50Gi
        mountPath: /var/lib/kubelet
        name: kubelet
    - name: nogpuworker
      replicas: 1
      storageClass: kubernetes-policy
      vmClass: best-effort-small

Support New Package Based Framework

Another capability in the new TKRs is standardization on a package based format.  You can see the packages included in the TKR with :  kubectl get tkr -o yaml |grep -A 10

/home/ubuntu: kubectl get tkr -o yaml |grep -A 10 bootstrapPackages
    bootstrapPackages:
    - name: antrea.tanzu.vmware.com.1.5.3+tkg.2-zshippable
    - name: vsphere-pv-csi.tanzu.vmware.com.2.6.0+vmware.1-tkg.1-zshippable
    - name: vsphere-cpi.tanzu.vmware.com.1.23.1+vmware.1-tkg.2-zshippable
    - name: kapp-controller.tanzu.vmware.com.0.41.2+vmware.1-tkg.1-zshippable
    - name: guest-cluster-auth-service.tanzu.vmware.com.1.0.0+tkg.2-zshippable
    - name: metrics-server.tanzu.vmware.com.0.6.1+vmware.1-tkg.3-zshippable
    - name: secretgen-controller.tanzu.vmware.com.0.11.0+vmware.2-tkg.1-zshippable
    - name: pinniped.tanzu.vmware.com.0.12.1+vmware.2-tkg.2-zshippable
    - name: capabilities.tanzu.vmware.com.0.28.0-dev-19-g0e3cb04b+vmware.1
    - name: calico.tanzu.vmware.com.3.22.1+vmware.1-tkg.2-zshippable

Notice that this TKR includes both antrea and calico CNI packages.  So a user could choose to deploy their cluster using either of these for overlay networking.   TKG 2 supports the enablement of these packages at deployment time but also provides a new capability for package management after deployment through the Tanzu CLI and Tanzu Package Repository

The new Tanzu Kubernetes Release (TKR) provides flexibility and cross infrastructure standardization that was not available with the legacy TKR.  

 

Filter Tags

Modern Applications Tanzu Kubernetes Grid vSphere with Tanzu Kubernetes Tanzu Kubernetes Grid Blog Deep Dive Advanced Manage

Michael West

Read More from the Author

Michael is a Staff Technical Product Manager in the Cloud Platform Business Unit and has been with VMware for 13 years. He is focused on technologies like Kubernetes that operationalize container based applications at scale. When not ruining his eyesight while staring at small font CLIs, he enjoys fitness training, stand up paddle boarding and his Staffordshire Bull Terrier - Elvis.