vSphere with Tanzu Private Registry Support

The Problem

Private registries with self-signed certificates (or certs signed by internal CAs) would generate x509: certificate signed by unknown authority during an image pull, meaning that the requester (a TKG cluster worker node) does not trust the certificate presented by the registry causing image pulls to fail.

The Solution

To use a private registry with Tanzu Kubernetes Clusters, configure the Tanzu Kubernetes Grid Service TkgServiceConfiguration with one or more self-signed certificates using the new additionalTrustedCAs field:

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TkgServiceConfiguration
metadata:
  name: tkg-service-configuration
spec:
  defaultCNI: antrea
  trust:
    additionalTrustedCAs:
      - name: first-cert-name
        data: base64-encoded string of a PEM encoded public cert
      - name: second-cert-name
        data: base64-encoded string of a PEM encoded public cert

TLS certificates will be applied to new clusters the next time a cluster is created but does not apply to existing clusters and will not trigger a rolling update - the next time you update an existing TKC the CA certs will be added.

An Example Use Case

I have deployed Harbor to another TKG cluster with helm using the following yaml and command:

expose:
  type: loadBalancer
  tls:
    certSource: auto
    auto:
      commonName: "core.harbor.labs.satm.eng.vmware.com"

externalURL: https://core.harbor.labs.satm.eng.vmware.com

persistence:
  enabled: true
  resourcePolicy: ""
  persistentVolumeClaim:
    registry:
      storageClass: "vsan-default-storage-policy"
    chartmuseum:
      storageClass: "vsan-default-storage-policy"
    jobservice:
      storageClass: "vsan-default-storage-policy"
    database:
      storageClass: "vsan-default-storage-policy"
    redis:
      storageClass: "vsan-default-storage-policy"
    trivy:
      storageClass: "vsan-default-storage-policy"
  imageChartStorage:
    type: filesystem
    filesystem:
      rootdirectory: /storage

harborAdminPassword: "P@ssw0rd"

This will make harbor deploy and generate a self-signed certificate and deploy nginx to front the registry, you can extract the certificate from the TKG cluster post-deployment and put it on your local filesystem with the following:

kubectl get secret harbor-harbor-nginx -o=jsonpath="{.data.ca\.crt}" > harbor-ca.crt 

We can take the harbor-ca.crt we extracted (it's already base64 encoded for us) an add it as an entry in our TkgServiceConfiguration (i'm calling this file tks.yaml):

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TkgServiceConfiguration
metadata:
  name: tkg-service-configuration
spec:
  defaultCNI: antrea
  trust:
    additionalTrustedCAs:
      - name: harbor-ca
        data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lRUHdpYUV4RjZsUzA2cGQzc2FsRWIyREFOQmdrcWhraUc5dzBCQVFzRkFEQVUKTVJJd0VBWURWUVFERXdsb1lYSmliM0l0WTJFd0hoY05NakV3TXpFd01UTTFOVFF6V2hjTk1qSXdNekV3TVRNMQpOVFF6V2pBVU1SSXdFQVlEVlFRREV3bG9ZWEppYjNJdFkyRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCCkR3QXdnZ0VLQW9JQkFRRG5Ta0ZCRTNvbk4yR0VOYU1TRWJRSUNzVmVUajJOQTBWRC9nNHdKZ3pwMVU5alc3VE8KNC9KK0JxNE9aS2Nia2NTWFcxYkNDeWNKKzNpQ2RIdFFqaVUvSnZZRHZhcGlBVnA0UmIvRE5lYnJyL0VaR2hZQQpUUjAybWpuS0hlVGdJVXBsTk5OM0haM0x2OE1xemxwem1OaVNKZWZCOFpqMWI0eElZS09LVWZPSzBJRjQrTERsCkFEUHJ0L1B4TmN3NDQ3RUI4QVpuay92YVpGN1F3ekNzY2E4cUlyUlFzZmk4Z0JCcEJDVlllMk4wM1dPWE9aK2kKMXZ1WnVvREh3NEgwWGNQVjhiNlZnVGsxWjMrak92MUF0UXBIUXllcExpMCtnVDdXb1VJTCt5anJ0dVJHeGpkaAp1R2FoRnVhajRCNmZ1RHo1eHRtM2ZyMy9WNFRUYVEwRk9pbjVBZ01CQUFHallUQmZNQTRHQTFVZER3RUIvd1FFCkF3SUNwREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXcKQXdFQi96QWRCZ05WSFE0RUZnUVUvWEJZUW5SSlJjeXZzWjQxdHozMlRrZ2ZUdnd3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFMTWlGa1RlL1BtZCtuazZzVHFGQmlNd2REWEVFeTAyL2tsMVhLZ0RTOURVenU2bmpseXljSzFaCitGY3JNUGFpNnYrTjBySzBYbmt0TmRuRkxpdS9ZL2JiZENHSURWQ3R2c2FuY29rZHUyZFZyZDdRZWFXdzMwV3MKY1RKK0lGb214YUNDTjZ1bGd6QXNsbGJVL3ZJYzg5cHEzOEZNN3I4TXpUcjhyVFpIVzdiY29vZThlcHJHTE9PegpiR3Uxb2lCbWpBSlNrV3owejBETFJ0TGFhRHFUY1hsN09YNTZIWjRJamVSWll5RDloMFNRR0VtMURkR1EvSmdECmRQdXhsa01wSnZuQWtLcVN4K2JTMVlIdTA1dHdyYkdDRk5EY2RCd0owYlRJV1lVeUI1Y3BXUEJvQkxxMUxQQ1kKUHVyKzRRSWpjb05JME1uSCtEaW1Lb0Rwc0RMN083TT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

If we apply this to our Supervisor cluster, any TKG clusters created after this point will automatically trust that CA cert - and thus, the registry:

kubectl config use-context $Supervisor_Cluster_IP
kubectl apply -f tks.yaml 

Conclusion

As you can see, adding arbitrary CA certs is now simple in vSphere with Tanzu and has wide applicability, but clearly this new functionality is very useful for those of you running internal registries either because of the new Docker Hub rate limiting rules that are enforced, or for other reasons.

Now you can assign either self signed CA certs or certs signed by an internal CA to TKG clusters at creation time, without having to manually edit the clusters post-deployment.