The Problem
Private registries with self-signed certificates (or certs signed by internal CAs) would generate x509: certificate signed by unknown authority
during an image pull, meaning that the requester (a TKG cluster worker node) does not trust the certificate presented by the registry causing image pulls to fail.
The Solution
To use a private registry with Tanzu Kubernetes Clusters, configure the Tanzu Kubernetes Grid Service TkgServiceConfiguration
with one or more self-signed certificates using the new additionalTrustedCAs
field:
apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TkgServiceConfiguration
metadata:
name: tkg-service-configuration
spec:
defaultCNI: antrea
trust:
additionalTrustedCAs:
- name: first-cert-name
data: base64-encoded string of a PEM encoded public cert
- name: second-cert-name
data: base64-encoded string of a PEM encoded public cert
TLS certificates will be applied to new clusters the next time a cluster is created but does not apply to existing clusters and will not trigger a rolling update - the next time you update an existing TKC the CA certs will be added.
An Example Use Case
I have deployed Harbor to another TKG cluster with helm using the following yaml and command:
expose:
type: loadBalancer
tls:
certSource: auto
auto:
commonName: "core.harbor.labs.satm.eng.vmware.com"
externalURL: https://core.harbor.labs.satm.eng.vmware.com
persistence:
enabled: true
resourcePolicy: ""
persistentVolumeClaim:
registry:
storageClass: "vsan-default-storage-policy"
chartmuseum:
storageClass: "vsan-default-storage-policy"
jobservice:
storageClass: "vsan-default-storage-policy"
database:
storageClass: "vsan-default-storage-policy"
redis:
storageClass: "vsan-default-storage-policy"
trivy:
storageClass: "vsan-default-storage-policy"
imageChartStorage:
type: filesystem
filesystem:
rootdirectory: /storage
harborAdminPassword: "P@ssw0rd"
This will make harbor deploy and generate a self-signed certificate and deploy nginx to front the registry, you can extract the certificate from the TKG cluster post-deployment and put it on your local filesystem with the following:
kubectl get secret harbor-harbor-nginx -o=jsonpath="{.data.ca\.crt}" > harbor-ca.crt
We can take the harbor-ca.crt
we extracted (it's already base64
encoded for us) an add it as an entry in our TkgServiceConfiguration
(i'm calling this file tks.yaml
):
apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TkgServiceConfiguration
metadata:
name: tkg-service-configuration
spec:
defaultCNI: antrea
trust:
additionalTrustedCAs:
- name: harbor-ca
data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lRUHdpYUV4RjZsUzA2cGQzc2FsRWIyREFOQmdrcWhraUc5dzBCQVFzRkFEQVUKTVJJd0VBWURWUVFERXdsb1lYSmliM0l0WTJFd0hoY05NakV3TXpFd01UTTFOVFF6V2hjTk1qSXdNekV3TVRNMQpOVFF6V2pBVU1SSXdFQVlEVlFRREV3bG9ZWEppYjNJdFkyRXdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCCkR3QXdnZ0VLQW9JQkFRRG5Ta0ZCRTNvbk4yR0VOYU1TRWJRSUNzVmVUajJOQTBWRC9nNHdKZ3pwMVU5alc3VE8KNC9KK0JxNE9aS2Nia2NTWFcxYkNDeWNKKzNpQ2RIdFFqaVUvSnZZRHZhcGlBVnA0UmIvRE5lYnJyL0VaR2hZQQpUUjAybWpuS0hlVGdJVXBsTk5OM0haM0x2OE1xemxwem1OaVNKZWZCOFpqMWI0eElZS09LVWZPSzBJRjQrTERsCkFEUHJ0L1B4TmN3NDQ3RUI4QVpuay92YVpGN1F3ekNzY2E4cUlyUlFzZmk4Z0JCcEJDVlllMk4wM1dPWE9aK2kKMXZ1WnVvREh3NEgwWGNQVjhiNlZnVGsxWjMrak92MUF0UXBIUXllcExpMCtnVDdXb1VJTCt5anJ0dVJHeGpkaAp1R2FoRnVhajRCNmZ1RHo1eHRtM2ZyMy9WNFRUYVEwRk9pbjVBZ01CQUFHallUQmZNQTRHQTFVZER3RUIvd1FFCkF3SUNwREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXcKQXdFQi96QWRCZ05WSFE0RUZnUVUvWEJZUW5SSlJjeXZzWjQxdHozMlRrZ2ZUdnd3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFMTWlGa1RlL1BtZCtuazZzVHFGQmlNd2REWEVFeTAyL2tsMVhLZ0RTOURVenU2bmpseXljSzFaCitGY3JNUGFpNnYrTjBySzBYbmt0TmRuRkxpdS9ZL2JiZENHSURWQ3R2c2FuY29rZHUyZFZyZDdRZWFXdzMwV3MKY1RKK0lGb214YUNDTjZ1bGd6QXNsbGJVL3ZJYzg5cHEzOEZNN3I4TXpUcjhyVFpIVzdiY29vZThlcHJHTE9PegpiR3Uxb2lCbWpBSlNrV3owejBETFJ0TGFhRHFUY1hsN09YNTZIWjRJamVSWll5RDloMFNRR0VtMURkR1EvSmdECmRQdXhsa01wSnZuQWtLcVN4K2JTMVlIdTA1dHdyYkdDRk5EY2RCd0owYlRJV1lVeUI1Y3BXUEJvQkxxMUxQQ1kKUHVyKzRRSWpjb05JME1uSCtEaW1Lb0Rwc0RMN083TT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
If we apply this to our Supervisor cluster, any TKG clusters created after this point will automatically trust that CA cert - and thus, the registry:
kubectl config use-context $Supervisor_Cluster_IP
kubectl apply -f tks.yaml
Conclusion
As you can see, adding arbitrary CA certs is now simple in vSphere with Tanzu and has wide applicability, but clearly this new functionality is very useful for those of you running internal registries either because of the new Docker Hub rate limiting rules that are enforced, or for other reasons.
Now you can assign either self signed CA certs or certs signed by an internal CA to TKG clusters at creation time, without having to manually edit the clusters post-deployment.