]

Solution

  • Modern Applications

Category

  • VCF Operational Guidance

Product

  • Cloud Foundation

Kubernetes Sizing and Planning on VMware Cloud Foundation

Cloud Foundation Kubernetes Sizing Guide

A crucial step in planning your Kubernetes deployments on VMware Cloud Foundation is sizing the underlying infrastructure.  

 

There are multiple components that need to be considered when sizing workload domains used to host Kubernetes clusters.

 

  • vCenter Server deployment size

  • NSX Manager deployment size

  • NSX Edge Transport Node deployment size

  • Supervisor Cluster node deployment size

 

It's important to note that some of these components will run in the Management domain.  Others will run in the workload domain.  As such, sizing must be considered for both the new domain and the Management Domain.

vCenter Server Sizing

vCenter Server Sizing

With the Cloud Foundation standard architecture, the vCenter Server instances runs in the management domain.  By default, vCenter Servers are deployed using the medium form factor.  Verify there is adequate capacity in the management domain for the new vCenter Server instance.

 

vCenter Server Footprint in the Management Domain

Size

Total vCPU

Total Memory

Total Storage

Medium

8 vCPU

28GB

700GB

Notes:  

  1. Starting with VCF 4.0.1 the  VCF API can be be used to customize the vCenter Server size when deploying new domains.

  2. When using the Cloud Foundation consolidated architecture, there is a one domain with a single vCenter Server instance.  This instance is deployed by the Cloud Builder during bring-up and the form factor set in the input parameters file.

NSX Manager Sizing

NSX Manager Sizing

Workload domains can be deployed with a dedicated NSX-T fabric or as part of a shared NSX-T fabric.  When deploying a new NSX-T fabric, additional capacity is needed in the Management Domain for the three NSX Manager nodes.  

 

When deploying a domain with a new NSX-T fabric, verify there is adequate capacity in the management domain for the three additional NSX-T Manager nodes.  By default, the NSX Manager nodes are deployed using the medium  form factor.  

 

NSX Manager Footprint in the Management Domain

Size

Total vCPU

Total Memory

Total Storage

Medium (3 nodes)

3 @ 6 vCPU = 18vCPU

3 @ 24GB = 72GB

3 @ 200GB = 600GB

Notes:

  1. Starting with VCF 4.0.1 the  VCF API can  be be used to customize the NSX Manager node size when deploying new domains.

  2. When using the Cloud Foundation consolidated architecture, all workloads share a common NSX-T fabric that gets deployed by Cloud Builder during bring-up.  The NSX Manager node form factor is set in the input parameters file.

  3. In small lab configurations the API can be used to deploy a single NSX Manager instance.  Running a single-node NSX Manger is not a supported implementation, but can be used for demos and small lab deployments.

NSX Edge Transport Node Sizing

NSX Edge Transport Node Sizing

Deploying an NSX Edge Cluster is a a prerequisite for enabling Kubernetes on a Cloud Foundation domain.  NSX Edge Transport nodes run in the workload domain.

 

NSX-T is automatically deployed and configured with each new domain during domain creation.  After a new domain has been created, use the SDDC Manager to deploy a pair of Edge Transport Nodes and configure an NSX Edge Cluster.  NSX Edge Clusters enable logical routing,  NAT, and load balancing services for the domain.  

 

Domains used to host Kubernetes clusters require two Edge Transport Nodes deployed as an active/active pair using the large form factor.  Edge Transport Nodes run in the vSphere cluster backing the workload domain.  

 

NSX Edge Custer Footprint in the Workload Domain

Size

Total vCPU

Total Memory

Total Storage

Large (2 nodes)

2 @32 vCPU = 64vCPU

2 @ 8GB = 16GB

2 @ 200GB = 400GB

Notes:

  1. The SDDC Manager UI supports deploying two-node NSX Edge Clusters.    After the Edge Cluster has been created, you can expand it by adding additional nodes (up to NSX configuration maximums) using the NSX Manager.

  2. When deploying an NSX-T Edge Cluster for use with Kubernetes, choose the "Workload Management" use case in the SDDC Manager UI. 

Supervisor Cluster Node Sizing

Supervisor Cluster Node Sizing

When Workload Management is enabled on a vSphere Cluster three special purpose virtual machines are deployed.  These VMs, instantiate the Kubernetes control plane.  While the number of control plane VMs is not configurable, the admin can choose the size.

There are four possible form factors for the Supervisor Control Plane VMs:  Tiny, Small, Medium and Large.  The UI provides general guidance on the number of pods targeted for each deployment type.

Supervisor Control Plane VM Sizing

Remember there are always three control plane VMs, so you will need to multiple the numbers shown in the UI by three.    The table below shows the total footprint for the Supervisor Control Plane VMs.  

 

Total Supervisor Cluster Footprint in the Workload Domain

Size

Total vCPU

Total Memory

Total Storage

Tiny

6 vCPU

24GB

48GB

Small

12 vCPU

48GB

48GB

Medium

24 vCPU

72GB

48GB

Large

48 vCPU

96GB

48GB

Filter Tags

  • Modern Applications
  • VCF Operational Guidance
  • Cloud Foundation