Kubernetes Sizing and Planning on VMware Cloud Foundation
Cloud Foundation Kubernetes Sizing Guide
A crucial step in planning your Kubernetes deployments on VMware Cloud Foundation is sizing the underlying infrastructure.
There are multiple components that need to be considered when sizing workload domains used to host Kubernetes clusters.
-
vCenter Server deployment size
-
NSX Manager deployment size
-
NSX Edge Transport Node deployment size
-
Supervisor Cluster node deployment size
It's important to note that some of these components will run in the Management domain. Others will run in the workload domain. As such, sizing must be considered for both the new domain and the Management Domain.
vCenter Server Sizing
vCenter Server Sizing
With the Cloud Foundation standard architecture, the vCenter Server instances runs in the management domain. By default, vCenter Servers are deployed using the medium form factor. Verify there is adequate capacity in the management domain for the new vCenter Server instance.
vCenter Server Footprint in the Management Domain
|
Size |
Total vCPU |
Total Memory |
Total Storage |
|
Medium |
8 vCPU |
28GB |
700GB |
Notes:
-
Starting with VCF 4.0.1 the VCF API can be be used to customize the vCenter Server size when deploying new domains.
-
When using the Cloud Foundation consolidated architecture, there is a one domain with a single vCenter Server instance. This instance is deployed by the Cloud Builder during bring-up and the form factor set in the input parameters file.
NSX Manager Sizing
NSX Manager Sizing
Workload domains can be deployed with a dedicated NSX-T fabric or as part of a shared NSX-T fabric. When deploying a new NSX-T fabric, additional capacity is needed in the Management Domain for the three NSX Manager nodes.
When deploying a domain with a new NSX-T fabric, verify there is adequate capacity in the management domain for the three additional NSX-T Manager nodes. By default, the NSX Manager nodes are deployed using the medium form factor.
NSX Manager Footprint in the Management Domain
|
Size |
Total vCPU |
Total Memory |
Total Storage |
|
Medium (3 nodes) |
3 @ 6 vCPU = 18vCPU |
3 @ 24GB = 72GB |
3 @ 200GB = 600GB |
Notes:
-
Starting with VCF 4.0.1 the VCF API can be be used to customize the NSX Manager node size when deploying new domains.
-
When using the Cloud Foundation consolidated architecture, all workloads share a common NSX-T fabric that gets deployed by Cloud Builder during bring-up. The NSX Manager node form factor is set in the input parameters file.
-
In small lab configurations the API can be used to deploy a single NSX Manager instance. Running a single-node NSX Manger is not a supported implementation, but can be used for demos and small lab deployments.
NSX Edge Transport Node Sizing
NSX Edge Transport Node Sizing
Deploying an NSX Edge Cluster is a a prerequisite for enabling Kubernetes on a Cloud Foundation domain. NSX Edge Transport nodes run in the workload domain.
NSX-T is automatically deployed and configured with each new domain during domain creation. After a new domain has been created, use the SDDC Manager to deploy a pair of Edge Transport Nodes and configure an NSX Edge Cluster. NSX Edge Clusters enable logical routing, NAT, and load balancing services for the domain.
Domains used to host Kubernetes clusters require two Edge Transport Nodes deployed as an active/active pair using the large form factor. Edge Transport Nodes run in the vSphere cluster backing the workload domain.
NSX Edge Custer Footprint in the Workload Domain
|
Size |
Total vCPU |
Total Memory |
Total Storage |
|
Large (2 nodes) |
2 @32 vCPU = 64vCPU |
2 @ 8GB = 16GB |
2 @ 200GB = 400GB |
Notes:
-
The SDDC Manager UI supports deploying two-node NSX Edge Clusters. After the Edge Cluster has been created, you can expand it by adding additional nodes (up to NSX configuration maximums) using the NSX Manager.
-
When deploying an NSX-T Edge Cluster for use with Kubernetes, choose the "Workload Management" use case in the SDDC Manager UI.
Supervisor Cluster Node Sizing
Supervisor Cluster Node Sizing
When Workload Management is enabled on a vSphere Cluster three special purpose virtual machines are deployed. These VMs, instantiate the Kubernetes control plane. While the number of control plane VMs is not configurable, the admin can choose the size.
There are four possible form factors for the Supervisor Control Plane VMs: Tiny, Small, Medium and Large. The UI provides general guidance on the number of pods targeted for each deployment type.
Remember there are always three control plane VMs, so you will need to multiple the numbers shown in the UI by three. The table below shows the total footprint for the Supervisor Control Plane VMs.
Total Supervisor Cluster Footprint in the Workload Domain
|
Size |
Total vCPU |
Total Memory |
Total Storage |
|
Tiny |
6 vCPU |
24GB |
48GB |
|
Small |
12 vCPU |
48GB |
48GB |
|
Medium |
24 vCPU |
72GB |
48GB |
|
Large |
48 vCPU |
96GB |
48GB |