VMware Enterprise PKS with Multiple Availability Zones on VMware vSAN

Executive Summary

Business Case

Digital transformation is driving a new application development and deployment approach called cloud-native. Cloud-native applications empower your developers by providing the resources and environments they need to innovate and deliver applications faster on-premises or in the cloud. Leverage containers, cloud resources, and automation without compromising security and compliance. 

VMware Enterprise PKS enables enterprises to deploy and consume container services with production-grade Kubernetes orchestration. It provides a comprehensive solution for enterprises developing and deploying cloud-native applications.

VMware vSAN™ supports hybrid and all-flash deployment modes. With vSAN all-flash deployments, users can benefit from high performance with less time spent managing and tuning performance, as well as cost savings. The space saving features of vSAN such as deduplication and compression can compound these savings further. The Storage Policy Based Management (SPBM) provided with vSAN gives users flexibility to define policies on demand and ease the management of container storage. Data services such as snapshots, cloning, encryption, deduplication, compression are available at container-volume level granularity. A container’s volume is backed by a VMDK on vSAN, which is associated with a storage policy. This gives developers the flexibility to manage container storage in a much finer granularity.

vSAN, as a shared storage system for vSphere, provides a uniform management plane for container storage. The deep integration between VMware Enterprise PKS and vSAN means that developers can consume storage as code with freedom by abstracting the complexity of the underlying storage infrastructure. vSAN’s SPBM offers users flexibility to define policies on demand in VMware vCenter® and delivers ease of management of storage for containers.

With Project Hatchway/Cloud Native Storage and vSAN services, cloud-native applications benefit from hyperconverged storage and compute as well as seamless application failover and rapid recovery. Advanced storage features, such as deduplication, compression and high availability, are all part of the core solution.

Solution Overview

This solution is a showcase of using VMware vSAN as a platform for deploying VMware Enterprise PKS in a vSphere environment. All storage management moves into a single software stack, thus taking advantage of the security, operational simplicity, and cost-effectiveness of vSAN in production environments. Workloads can be easily migrated from bare-metal configurations to a modern, dynamic, and consolidated HCI solution based on vSAN. vSAN is natively integrated with vSphere, and helps to provide smarter solutions to reduce the design and operational burden of a data center. 

This solution focuses on the architecture of multiple Availability Zones (Multi-AZ) in Kubernetes (K8s).

Key Results

The reference architecture:

  • Provides the solution architecture for deploying PKS with Multi-AZ mode in a vSAN cluster for production.
  • Uses the NSX-T network.
  • Uses vRealize Operations for container monitoring and NSX-T monitoring.
  • Uses vRealize Log Insight for log analyzing.
  • Measures performance when running PKS in a vSAN cluster, to the extent of the testing and cluster size described.
  • Measures performance of the workloads running atop of PKS in a vSAN cluster.
  • Performs operation testing and validates the scalability of running PKS in vSAN.
  • Identifies steps required to ensure resiliency and availability against various failures.
  • Provides best practice guides. 

Introduction

Purpose

This solution illustrates how PKS can be run in a vSphere environment backed by vSAN and provides operation guidelines and testing results based on parameter variations of mixed workloads.   

Scope

The reference architecture covers the following testing scenarios:

  • Deploy Kubernetes clusters with Multi-AZ using PKS management plane
  • Scale out Kubernetes clusters with Multi-AZ using PKS management plane
  • Mixed workloads performance of MongoDB and MySQL with Multi-AZ in the Kubernetes cluster deployed by PKS
  • Operation testing
  • Scalability testing 
  • Resiliency and availability testing 

Audience

This paper is intended for cloud-native application administrators and storage architects involved in planning, designing, or administering of PKS on vSAN.

Technology Overview

This section provides an overview of the technologies used in this solution:

  • VMware vSphere® 6.7 Update 1
  • VMware vSAN 6.7 Update 1
  • VMware Enterprise PKS 1.4
  • VMware vRealize Operations Management Pack for Container
  • VMware vRealize Operations Management Pack for NSX-T™ 
  • VMware vRealize Log Insight with NSX-T Content Pack
  • MongoDB 3.6
  • MySQL 5.7

VMware vSphere 6.7 Update 1

VMware vSphere 6.7 is the infrastructure for next-generation applications. It provides a powerful, flexible, and secure foundation for business agility that accelerates the digital transformation to cloud computing and promotes success in the digital economy.  

vSphere 6.7 supports both existing and next-generation applications through its:  

  • Simplified customer experience for automation and management at scale 
  • Comprehensive built-in security for protecting data, infrastructure, and access 
  • Universal application platform for running any application anywhere 

With vSphere 6.7, customers can run, manage, connect, and secure their applications in a common operating environment, across clouds and devices.

VMware vSAN 6.7 Update 1

VMware vSAN, the market leader in Hyperconverged Infrastructure (HCI), enables low-cost and high-performance next-generation HCI solutions, converges traditional IT infrastructure silos onto industry-standard servers and virtualizes physical infrastructure to help customers easily evolve their infrastructure without risk, improve TCO over traditional resource silos, and scale to tomorrow with support for new hardware, applications, and cloud strategies. The natively integrated VMware infrastructure combines radically simple VMware vSAN storage, the market-leading VMware vSphere Hypervisor, and the VMware vCenter Server® unified management solution all on the broadest and deepest set of HCI deployment options. 

See VMware vSAN documentation for more information. 

VMware NSX-T Data Center 2.3

NSX-T Data Center is focused on emerging application frameworks and architectures that have heterogeneous endpoints and technology stacks. In addition to vSphere hypervisors, these environments include other hypervisors such as KVM, containers, and bare metal.

NSX-T Data Center is designed for management, operations, and consumption by development organizations. NSX-T Data Center allows IT and development teams to choose the technologies best suited for their applications. 

You can read the NSX-T Data Center document for more information.

VMware Enterprise PKS 1.4

VMware Enterprise PKS is a purpose-built container solution to operationalize Kubernetes for multicloud enterprises and service providers. It significantly simplifies the deployment and management of Kubernetes clusters with day 1 and day 2 operations support. With hardened production-grade capabilities, VMware Enterprise PKS takes care of your container deployments from the application layer all the way to the infrastructure layer.

VMware Enterprise PKS is built in with critical production capabilities such as high availability, autoscaling, health-checks, as well as self-healing and rolling upgrades for Kubernetes clusters. With constant compatibility to GKE, VMware Enterprise PKS provides the latest stable Kubernetes release so developers have the latest features and tools available to them. It also integrates with VMware NSX-T for advanced container networking including micro-segmentation, ingress controller, load balancing and security policy. Through an integrated private registry, VMware Enterprise PKS secures container image via vulnerability scanning, image signing and auditing. 

VMware Enterprise PKS exposes Kubernetes in its native form without adding any layers of abstraction or proprietary extensions, which lets developers use the native Kubernetes CLI that they are most familiar with. VMware Enterprise PKS can be easily deployed and operationalized via Pivotal Operations Manager, which allows a common operating model to deploy VMware Enterprise PKS across multiple IaaS abstractions like vSphere, Google Cloud Platform (GCP), and Amazon Web Services (AWS) EC2.

VMware vRealize® Operations Management Pack for Container Monitoring

With VMware vRealize Operations Management Pack for Container Monitoring, virtual administrators can get complete Kubernetes topology of Namespaces, Clusters, Replica Sets, Nodes, Pods, and Containers for monitoring Kubernetes clusters. The Out of the box dashboard not only provides an overview of Kubernetes ecosystem but also troubleshoots by highlighting the Key Performance Index and alerts for various objects pertaining to Kubernetes clusters that are monitored. This management pack extends the monitoring capability of vRealize Operations Manager to provide insights to the Kubernetes clusters to the Virtual Infrastructure administrator. This version of management pack enables auto discovery of Kubernetes clusters available in VMware Enterprise PKS environment.

VMware vRealize Operations Management Pack for NSX-T™

VMware vRealize Operations Management Pack for VMware NSX-T provides inventory monitoring of the following NSX-T resources:

  • Controller clusters
  • Edge clusters
  • Logical routers
  • Transport zones and Transport nodes
  • Load balancers
  • Firewalls
  • Management services

In addition to monitoring the above mentioned resources, VMware vRealize Operations Management Pack for VMware NSX-T provides file system usage for individual nodes in the controller cluster, transmitted/received bytes and packet drop metrics for the ethernet interfaces. There are more than 10 out of the box alerts related to the above mentioned monitored resources.

VMware vRealize Log Insight with NSX-T Content Pack

Logs are the best source of information for troubleshooting issues, analyzing changes and obtaining runtime operational data. NSX-T has a consistent logging infrastructure and the logs are parsed and represented in the Log Insight Content Pack. The NSX-T content pack will enable easy onboarding and adoption of the product by Network and Operation teams and will provide consistent monitoring and troubleshooting capabilities.

The NSX-T Log Insight Content Pack provides the collection, consolidation and correlation of NSX log data. This content pack contains 7 dashboards and 52 widgets for representing the NSX-T logs graphically making them more intuitive and actionable. NSX-T admins can view issues related to communication between different NSX components. The widgets provide information on logs for different networking elements such as logical switch, logical routers and distributed firewalls. The logs include audit information, errors, configuration and traffic patterns. Log Insight Content Pack for NSX-T is supported by VMware and is designed for continuous monitoring by Network Engineering and Operations team in an organization.

MongoDB 3.6

MongoDB is a document-oriented database. The data structure is composed of field and value pairs. MongoDB documents are similar to JSON objects. The values of fields may include other documents, arrays, and arrays of documents.

The advantages of using documents are:

  • Documents (for example, objects) correspond to native data types in many programming languages.
  • Embedded documents and arrays reduce the need for expensive joins.
  • Dynamic schema supports fluent polymorphism.

For more key features of MongoDB, refer to Introduction to MongoDB.

MySQL 5.7

MySQL is the most popular open source database system which enables the cost-effective delivery of reliable, high-performance and scalable web-based and embedded database applications. It is an integrated transaction safe, ACID-compliant database with full commit, rollback, crash recovery, and row-level locking capabilities. MySQL delivers the ease of use, scalability, and high performance, as well as a full suite of database drivers and visual tools to help developers and DBAs build and manage their business-critical MySQL applications. 

Configuration

This section introduces the resources and configurations: • Solution architecture • Network architecture • Hardware resources • Software resources • VM configurations • Test tool and workload

Solution Architecture

We used a separate and shared 4-hosts vSAN cluster for management purpose. The management cluster was shared between different workloads, not limited to the VMware Enterprise PKS computing cluster. The management and infrastructure virtual machines such as Windows Active directory Server, DNS server, NTP server and other necessary servers are residing on the management cluster. We also deployed vRealize Automation, vRealize Operations and vRealize Log insight in the management cluster. They were used in a shared manner across different workload domains and used for different purposes.

This reference architecture is focused on VMware Enterprise PKS and we would be using just one VMware Enterprise PKS cluster for illustration in the following parts of this reference architecture.

Specifically for VMware Enterprise PKS deployment, we installed VMware vRealize® Operations Management Pack for Container Monitoring, VMware vRealize Operations Management Pack for NSX-T™ and VMware vRealize Log Insight with NSX-T Content Pack in vRealize Operations and vRealize Log insight in the management cluster for VMware Enterprise PKS monitoring and analyzing.

 

We used a 6-node vSphere and vSAN cluster as the VMware Enterprise PKS computing cluster, and deployed the necessary virtual machines on it. The VMware Enterprise PKS computing cluster architecture is depicted in Figure 1.

Figure 1. High Level Architecture of the VMware Enterprise PKS on vSAN Reference Architecture

In the reference architecture, we had 3 availability zones for PKS. To achieve Multi-AZ in PKS, we did the following configuration:

  • Created 3 vSAN Fault Domains. ESXi-1 and ESXi-2 were in vSAN Fault Domain 1. ESXi-3 and ESXi-4 were in vSAN Fault Domain 2. ESXi-5 and ESXi-6 were in vSAN Fault Domain 3.
  • Create 3 DRS Host Groups. The host groups must be grouped exactly as the vSAN fault domains. ESXi-1 and ESXi-2 were in Host Group 1. ESXi-3 and ESXi-4 were in Host Group 2. ESXi-5 and ESXi-6 were in Host Group 3.

The virtual machines are divided into four groups:

  • Infrastructure VMs were residing on the management cluster, they are not explicitly shown in this figure. They included:
  • vCenter: Manages this vSphere and vSAN cluster.
  • DNS Server: Acts as a DNS forwarder as well as contains the local DNS entries for NSX Manager, Pivotal Operations Manager, and Harbor.
  • VMware Enterprise PKS Client: Runs the ‘pkscli’ and ‘kubectl’ binaries for managing the VMware Enterprise PKS environment.
  • NSX-T VMs were residing on the management cluster, they are not explicitly shown in this figure. The NSX-T virtual machines contain NSX-T Manger, NSX-T Edges, and NSX-T Controllers. These virtual machines are deployed based on the NSX-T installation guide: NSX-T documentation.
  • VMware Enterprise PKS VMs: The VMware Enterprise PKS management virtual machines are deployed based on the VMware Enterprise PKS installation guide: PKS documentation. When deploying the VMware Enterprise PKS management virtual machines, they are not using the DRS Host Group feature. Therefore, these virtual machines were placed in the cluster by DRS automatically and protected by vSphere HA.
  • Deployed Kubernetes Clusters.

These Kubernetes clusters are deployed by VMware Enterprise PKS. The virtual machines are grouped by each Kubernetes cluster. With Multi-AZ feature enabled, each Kubernetes cluster contains three virtual machines acting as the primary nodes. Each virtual machine was placed in one availability zone. Then we could deploy at least three worker nodes as application needs. We can deploy more than one Kubernetes clusters based on the vSAN cluster’s physical resources.

Network Architecture

Physical network connection

On each physical server, four Network Interface Card (NIC) ports are used. Figure 2 shows the NIC and physical switch connection architecture. 

NIC 1 and NIC 2 are connected to the NSX-T N-VDS of the overlay transport zone. NIC 3 and NIC 4 are connected to the NSX-T N-VDS of the VLAN transport zone. For the naming convenience, NIC 1 was configured as uplink-1, NIC 2 was configured as uplink-2 and so on.

  • All the VMKernels were migrated from standard vSwitch to N-VDS.
  • We used the uplink teaming policy to let Management VMKernel and VMware vSphere vMotion® VMKernel network use NIC 1 as active uplink and NIC 2 as standby uplink.
  • Meanwhile, we used a similar uplink teaming policy to let vSAN VMKernel network uses NIC 2 as active uplink and NIC 1 as standby uplink.
  • The N-VDS used for VLAN transport zone are not configured with any uplink teaming policy. 

MTU settings

NSX-T requires MTU to be at least 1600 for the physical switches and NICs.

vSAN supports MTU 1500 and MTU 9000 and the performance tests show that larger MTU settings can help vSAN improve throughput.

Based on the above requirements, we set the MTU to 9000 in the whole environment to reduce the physical network management complexity and pursue a higher performance.

Figure 2. NIC Ports and Switch Physical Connection Architecture

Hardware Resources

Table 1 shows the hardware resources used in this solution.

We used a vSAN cluster with 6 physical servers. Each ESXi Server in the vSAN cluster has the following configuration. 

Table 1. Hardware Resources per ESXi Server

PROPERTY

SPECIFICATION

Server

Quanta Cloud Technology D52B-1U

CPU and cores

2 sockets, 28 cores each of 2.2GHz with hyper-threading enabled

RAM

256GB

Network adapter

4 x 10Gb NIC ports

Disks

Cache-layer SSD: 2 x 1.6 TB Intel Optane SSD DC P4600 NVMe (controller included)

Capacity-layer SSD: 6 x 3.84TB Intel S4500

In our various vSAN reference architectures, we recommended vSAN cluster with at least 4 physical hosts. If there are only 3 hosts in a vSAN cluster, there will be no rebuild if a host fails. Therefore, some data may be in risk during the repairing windows. With vSAN cluster of 4 hosts, degraded data due to host failure can be rebuilt immediately as long as there is enough free space. In this reference architecture, we used 6 physical servers and divided them into 3 fault domains and 3 host groups. This was in order to meet the requirements of 3 availability zones in VMware Enterprise PKS. Besides, there was also the capability to rebuild the data if one of the hosts in a fault domain failed as long as there was enough free space.

Tailored for a hyperconverged solution, QuantaGrid D52B-1U features ultimate compute and storage density in a 1U platform. Quanta Cloud Technology’s (QCT) well-designed second-generation Purley Server Platform and marketing -leading virtualization software developed by VMware, delivering a reliable and confident choice for customers. With the careful validation process on vSAN ReadyNode by QCT and VMware on QuantaGrid D52B-1U, customers can rest assured of the solution reliability and focus on strategic and productive tasks.

QCT powerful server provides ultimate compute and storage density, flexible and scalable I/O options, and solid reliability making it excellent for diversified cloud-native applications workloads. The compute capability with IntelXeon Gold 6138 CPUs empower VMware vSAN by supporting a wide range of critical workloads. Enterprises that value performance can benefit from the cache layer using the Intel SSD Data Center Family with NVMe. By adopting Optane SSD DC P4800X as the cache layer in vSAN, we can deliver an extremely high-performance and reduce the transaction cost while running the write-intensive workload. Therefore, QuantaGrid D52B-1U is an optimum option for the next-generation workload.

Visit the QCT website for more details of Quanta Cloud Technology and Quanta hardware platform.

Software Resources

Table 2 shows the software resources used in this solution.

Table 2. Software Resources

Software

Version

Purpose

VMware vCenter Server and ESXi 

6.7 Update 1 

(vSAN 6.7 Update 1 is included)

ESXi Cluster to host virtual machines and provide vSAN Cluster. VMware vCenter Server provides a centralized platform for managing VMware vSphere environments.

VMware vSAN

6.7 Update 1

Solution for HCI.

Ubuntu

16.04

Ubuntu 16.04 is used as the guest operating system of the DNS server, testing clients, etc.

NSX-T Data Center

2.3

NSX-T 2.3

Pivotal Operations Manager

2.5

Pivotal Operations Manager is the central place for deploying and managing Pivotal products such as VMware Enterprise PKS.

VMware Enterprise PKS

1.4

VMware Enterprise PKS 1.4

Harbor

1.7.5

Harbor is used for deploying a private enterprise-class container image repository.

Note: For Pivotal Operations Manager, a minimum version of 2.5.0 is required for using the DRS Host Group feature when deploying VMware Enterprise PKS Multi-AZ feature.

VM Configurations

We used the virtual machine settings as the base configuration as shown in Table 3 and Table 4. The virtual machines are grouped by different categories: NSX-T virtual machines, VMware Enterprise PKS management virtual machines, and VMware Enterprise PKS deployed virtual machines.

For vm sizing, the rule  is that the aggregated CPU cores and memory should not exceed the physical resources to avoid contention. When calculating physical resources, we should count the physical cores before hyper-threading is taken into consideration. 

Table 3. NSX-T Virtual Machines Configuration

PROPERTY

vCPU

MEMORY(GB)

STORAGE (GB)

NSX-T Manager

8

32

140

NSX-T Controller

4

16

120

NSX-T Edge

8

16

120

The configuration of NSX-T Controller is for one instance of controller. We should properly size for the controller cluster since a controller cluster is recommended to have at least 3 controllers. The configuration of NSX-T Edge in the table is also for one edge instance.

Note: For NSX-T edge, we used the ‘large’ profile when deploying edge virtual machines. ‘Large’ edge instances are required for VMware Enterprise PKS deployment with NSX-T integration.

Table 4 shows the configuration of the VMware Enterprise PKS management plane virtual machines.

Table 4. VMware Enterprise PKS Management Plane Virtual Machines Configuration

PROPERTY

vCPU

MEMORY(GB)

STORAGE (GB)

Pivotal Operations Manager

1

8

160

Bosh director

4

16

3+100+100

VMware Enterprise PKS controller

2

8

3+16+100

Harbor

2

8

3+64+1000

We used 1,000GB for Harbor’s data disk. This data disk size could be set during the Harbor deployment configuration page in Pivotal Operations Manager. The data disk size is calculated by the stored number of images and size of images. In vSAN, a 1,000GB VMDK is striped into at least 4 components because the maximum size of a component in vSAN is 255GB. If the disk size of Harbor’s data is smaller than 255GB, consider increasing the stripe width of the VMDK to improve read and write performance of Harbor by changing its vSAN storage policy. 

Test Tool and Mixed Workloads

MongoDB and YCSB

For MongoDB, we used the YCSB tool to evaluate the performance. YCSBis a popular Java open-source specification, and program suite developed at Yahoo to compare the relative performance of variousNoSQLdatabases. Its workloads are used in various comparative studies of NoSQL databases. 

We used YCSB workload A as summarized:

  • Workload A (Update heavy workload): 50/50% mix of reads/writes

Some key configuration parameters are in Table 5.

Table 5. YCSB Parameter Settings Used in this Solution Testing

PROPERTY

SPECIFICATION

Record count

100,000,000

Operation count

50,000,000

Threads on each client

16

Write acknowledgement

Majority

MySQL and SysBench

In this solution, we used SysBench to measure the performance of MySQL cluster.

SysBench is a modular, cross platform and multithreaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load. The OLTP test mode is used in solution validation to benchmark a real MySQL database performance. 

In this solution validation, we used the SysBench OLTP library to populate MySQL test database, and generate workload for performance and protection testing. 

We used a containerized SysBench client described in docker hub. We needed to modify the parameters and hostnames in the yaml files to match our own testbed. Table 6 lists some key configuration parameters.

Table 6. SysBench Parameter Settings

PROPERTY

SPECIFICATION

Oltp_table_size

10,000,000

Oltp_tables_count

8

Threads on each client

8

Time

3,600

Solution Validation

Overview

We conducted an extensive validation to demonstrate vSAN as a persistent storage layer for VMware Enterprise PKS.

The solution goal was to prove the functionality, flexibility, and performance of vSAN when used as the persistent storage layer for VMware Enterprise PKS. We tested the rapidity of spawning a bunch of Kubernetes clusters in parallel. We also tested performance of the traditional database MySQL and a next-generation NoSQL database MongoDB.

This solution included the following tests:

  • Performance testing of Kubernetes cluster operations: This set of tests validated the performance of Kubernetes cluster operations by using VMware Enterprise PKS, such as cluster creation and scaling out.
  • Performance testing of MongoDB: This set of tests validated the basic performance of running MongoDB as pods in a Kubernetes cluster deployed by VMware Enterprise PKS. We also tested the scaling out performance of MongoDB by increasing the number of MongoDB clusters from 1 to 2 to 4.
  • Performance testing of MySQL: This set of tests validated the basic performance of running MySQL as pods in a Kubernetes cluster deployed by VMware Enterprise PKS. We also tested the scaling out performance of MySQL by increasing the number of MySQL clusters from 1 to 2 to 4.
  • Resiliency and availability testing: From different layers, we tested different failure types. The failure types included host failure, virtual machine failure, vSAN disk failure, and Kubernetes pod failure.

Result Collection and Aggregation

  • The test results are aggregated because each client instance produces its own output:
    • For MongoDB, the total throughput ops/sec is the sum-up of ops/sec in all clients and latencies are the average values of all client instances.
    • For MySQL, TPS (transactions per second) is the sum-up of ops/sec in all clients and latencies are the average values of all client instances.
  • If there are any abnormal results, we use vSAN Performance Service to monitor each level of performance in a top-down approach of the vSAN stack.

Configure VMware Enterprise PKS deployment with Multi-AZ

In this section, we show how to create vSAN fault domains, vSphere DRS Host Groups and create corresponding availability zones in Pivotal Operations Manager.

Create vSAN Fault Domains

In vSphere Web Client, we created 3 fault domains in the cluster ‘Configure’ tab. Record the ESXi hosts of each groups in following sections.

Figure 3. Status of vSAN Fault Domains

Create vSphere DRS Host Groups

In vSphere Web Client, we created 3 host groups. The ESXi hosts in each host groups must be the same as the vSAN fault domains.

Figure 4. Status of vSphere DRS Host Groups

Create VMware Enterprise PKS Availability Zones

In Pivotal Operations Manager, we created 3 availability zones. Each availability zone is corresponding to a host group. Figure 5 showed that the availability zone called ‘pks-comp-az1’ was mapping to host group ‘PKS-HG1’.

Figure 5. Created 3 Availability Zones for PKS Compute Cluster

Specify Availability Zone for Kubernetes Primary and Worker Nodes in VMware Enterprise PKS Configuration

In Pivotal Operations Manager, we can specify the availability zones for Kubernetes primary and worker nodes deployed by VMware Enterprise PKS when configuring the ‘plans’ in VMware Enterprise PKS configuration tab. Figure 6 showed that we used all 3 availability zones for both primary and worker nodes placement.

Figure 6. Specifying Availability Zone for Kubernetes Primary and Worker Nodes in VMware Enterprise PKS Configuration

Once the availability zones were correctly configured, we could use VMware Enterprise PKS to deploy Kubernetes clusters and deploy workloads with the Multi-AZ feature.

Configure VMware Enterprise PKS deployment with Multi-AZ

In this section, we show how to create vSAN fault domains, vSphere DRS Host Groups and create corresponding availability zones in Pivotal Operations Manager.

Kubernetes Workloads Deployment with Multi-AZ Feature

In this section, we take deploying MongoDB workloads as an example to illustrate the Kubernetes workloads deployment with Multi-AZ feature. Specifically for vSAN, we illustrate ‘storage class creation’, ‘persistent volume claim creation’ and other automations when running VMware Enterprise PKS with vSAN.

We used storage policy and persistent volume claim to demonstrate the easy management, automation, and integration of vSAN and VMware Enterprise PKS. For use of other operations such as persistent volume creation and manual binding, refer to VMware Enterprise PKS and Kubernetes user guide.

Deploy Kubernetes Clusters using VMware Enterprise PKS

After we successfully installed VMware Enterprise PKS and command line tools, we could deploy Kubernetes clusters for successive use. We managed (create, delete, resize) the Kubernetes clusters by using the ‘pks’ command line tool.

1. Firstly, we used the following command to create a Kubernetes cluster named “k8s-cluster-1”:

$ pks create-cluster k8s-cluster-1 -e k8s-cluster-1.pks.lab -p 2xlarge -n 3

Figure 7. Creating a Kubernetes Cluster by Using the ‘pks’ Command Line Tool

2. As depicted in the screen shot, we could use the ‘pks cluster k8s-cluster-1’ command to monitor the creating status of the cluster. The ‘Last Action State’ was “in progress” when the cluster was not ready as shown in Figure 8. After a while, the cluster was successfully created and the ‘Last Action State’ was shown as ‘succeeded’.

Figure 8. Monitoring the Creating Cluster Status

3. From Figure 8, we could see that the IP address ‘192.168.129.2’ was allocated as the Kubernetes primary IP.

We used ‘k8s-cluster-1.pks.lab’ as the custom domain name for this cluster. So after the Kubernetes primary IP was allocated, we could modify our own corporate DNS server to map ‘k8s-cluster-1.pks.lab’ to ‘192.168.129.2’. In the NSX-T management UI, we could view the corresponding load balancer, routers, and switches created by NSX-T. They were ending with the UUID of the cluster as shown in Figure 9, Figure 10, and Figure 11.

Figure 9. The Load Balancer Created by NSX-T

Figure 10. The Routers Created by NSX-T

Figure 11. The Switches Created by NSX-T

NSX-T created the corresponding components automatically. After creation, we could use the Kubernetes primary IP to point to the Kubernetes cluster and all containers are created later to the switches and routers automatically to gain network access.

Scale out Kubernetes Clusters using VMware Enterprise PKS

After we initially deployed a Kubernetes cluster, we could scale it out if the workloads grew.

We used the following command to increase the number of worker nodes to 4. The process was monitored in Figure 12.

$ pks resize k8s-cluster-1 -n 4

Figure 12. Scaling out the Kubernetes Cluster and Monitor the Resizing Status

After the operation was successfully completed, the ‘Last Action’ showed ‘UPDATE’ and the ‘Last Action State’ showed ‘succeeded’. VMware Enterprise PKS just added a new worker VM to the cluster and all the NSX-T components remained the same.

Create Storage Class

Kubernetes supports various vSphere storages such VMFS, NFS, and vSAN. For example, we can define a storage class for creating VMDK with ‘Eager Zeroed Thick’ in an NFS datastore:

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

   name: fast

provisioner: kubernetes.io/vsphere-volume

parameters:

     diskformat: zeroedthick

     datastore: NFSDatastore

If we intend to use a different datastore, we need to modify the ‘datastore’ section in the storage class definition and add the corresponding parameters such as storage policy in vSAN.

If we use vSAN as the datastore, we can obtain the vSAN advantages such as Deduplication and Compression, Checksum verification, user-defined Stripe Width, more flexible data protection method (RAID 1/RAID 5/RAID 6) and so on. What all users need to do is modifying the storage class definition and the rest of the management is automated. 

Tag-based placement allows an administrator to assign tags to different datastores and then create a storage policy that can restrict access to datastores based on tag. VMware vSAN supports the full capabilities of SPBM, other non-vSAN vSphere-compatible storage backends only support the tag-based placement capability of SPBM. Using the tag-based placement, a developer can call a SPBM policy from the K8s manifest using the storagePolicyName variable. When the K8s control plane schedules the pod for provisioning and requests a volume, the storage policy ensures that the volume can only be provisioned from datastores that match the tags defined in the specified storage policy. If using vSAN, the following parameters can be dynamically tuned for each pod/deployment via the K8s manifest: cacheReservation, diskStripes, forceProvisioning, hostFailuresToTolerate, iopsLimit, objectSpaceReservation.

So, vSAN is a better solution for VMware Enterprise PKS compared to other vSphere storage types.

 

In this section we described the procedures of creating a Kubernetes storage class using vSAN datastore in details.

We created the necessary Kubernetes storage class before we deployed the actual stateful sets. This storage class would be used in the stateful sets creation in later steps. 

1. Firstly, we defined the storage class yaml file called default-vsan-sc.yaml:

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

    name: default-vsan-sc

provisioner: kubernetes.io/vsphere-volume

parameters:

    diskformat: thin

    storagePolicyName: "vSAN Default Storage Policy"

    datastore: vsanDatastore

The configuration item in bold used “vSAN Default Storage Policy” defined in vCenter. We can check the predefined policies in vCenter.

Figure 13. Predefined vSAN Storage Policies in vCenter

2. We also created a vSAN storage policy called “sw2” that used a minimum number of data stripe width to 2. We could also create another Kubernetes storage class by using this “sw2” policy. Then the VMDKs created in Kubernetes can use this vSAN policy. We create another storage class yaml file called sw2-vsan-sc.yaml:

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

    name: sw2-vsan-sc

provisioner: kubernetes.io/vsphere-volume

parameters:

    diskformat: thin

    storagePolicyName: "sw2"

    datastore: vsanDatastore

3. We used “kubectl” to create the storage classes and checked their status.

Figure 14. Create Storage Classes and Check the Status

Persistent Volume Claim

After the corresponding Kubernetes storage classes were created, we can use persistent volume claims to create persistent VMDKs in vSAN to be consumed by containers.

For each MongoDB component, we used a Kubernetes stateful set. For example, we used the following yaml file to define MongoDB’s ConfigDB. This file is called mongodb-configdb-service.yml.

apiVersion: v1

kind: Service

metadata:

   name: mongodb-configdb-service

   labels:

     name: mongo-configdb

spec:

   ports:

   - port: 27017

     targetPort: 27017

   clusterIP: None

   selector:

     app: mongo-configdb

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

   name: mongod-configdb

spec:

   serviceName: mongodb-configdb-service

   selector:

     matchLabels:

       app: mongo-configdb

   replicas: 1

   template:

     metadata:

       labels:

         app: mongo-configdb

         tier: configdb

         replicaset: ConfigDBRepSet

     spec:

       affinity:

         podAntiAffinity:

          requiredDuringSchedulingIgnoredDuringExecution:

            - labelSelector:

                matchExpressions:

                  - key: "app"

                    operator: In

                    values:

                    - mongo-configdb

               topologyKey: "bosh.zone"

       terminationGracePeriodSeconds: 10

       containers:

         - name: harbor1.pks.lab/library/mongod-configdb-container

          image: mongo:3.6.9

          command:

            - "numactl"

            - "--interleave=all"

            - "mongod"

            - "--port"

            - "27017"

            - "--wiredTigerCacheSizeGB"

            - "0.25"

            - "--bind_ip"

            - "0.0.0.0"

            - "--configsvr"

            - "--replSet"

             - "ConfigDBRepSet"

          ports:

            - containerPort: 27017

          volumeMounts:

            - name: mongo-configdb-persistent-storage-claim

              mountPath: /data/db

   volumeClaimTemplates:

   - metadata:

       name: mongo-configdb-persistent-storage-claim

     spec:

       accessModes: [ "ReadWriteOnce" ]

       storageClassName: default-vsan-sc

       resources:

         requests:

          storage: 4Gi

We used the following Kubernetes command to create this stateful set:

$ kubectl create -f mongodb-configdb-service.yml

During the execution of this command, a VMDK of 4 GB was automatically created in the vsanDatatore. 

Figure 15. VMDK Creation and Attachment with Persistent Volume Claim

This VMDK used the ‘vSAN Default Storage Policy’ because we used the “default-vsan-sc” storage class in the persistent volume claim definition section. In the previous step, we created this “default-vsan-sc” storage class with the ‘vSAN Default Storage Policy’.

VMware Enterprise PKS automatically selected a worker in this Kubernetes cluster and attached this VMDK to the worker VM. In this way, this VMDK created by the persistent volume claim can be consumed by the stateful set.

Scale out the Stateful Set

In the previous step of creating the “ConfigDB” stateful set, we used “replicas=1” in the yaml file as an initial configuration point. In a production environment, it is recommended to use more than one replica to provide redundancy. We can easily scale out the stateful set’s replicas by using this command:

$ kubectl scale --replicas=2 statefulset/mongod-configdb

Figure 16. Status of Persistent Volume Claim after Scaling out the Stateful Set

After scaling the stateful set by running the above command, another persistent volume claim and VMDK of 4 GB were created. It used the same configuration as defined in the yaml file.

This persistent volume claim was automatically created and automatically attached to another worker VM in the Kubernetes cluster. It used the same “vSAN Default Storage Policy” as defined in the storage class. The only command that users need to run is the one-line ‘kubectl’ command shown above.

Place Kubernetes Pods to Different Availability Zones

With pre-defined multiple availability zones, we could use the Kubernetes affinity and anti-affinity rules to explicitly place different pods into different availability zones. For example, in the above mongodb-configdb-service.yml configuration file, there is a section defining the ‘podAntiAffinity’ rules:

     spec:

       affinity:

         podAntiAffinity:

           requiredDuringSchedulingIgnoredDuringExecution:

            - labelSelector:

                matchExpressions:

                  - key: "app"

                    operator: In

                    values:

                    - mongo-configdb

               topologyKey: "bosh.zone"

We could refer to the Kubernetes tutorial for detailed instructions about assigning pods to nodes.

In our ‘podAntiAffinity’ rule, we used the topologyKey named “bosh.zone”. The “bosh.zone” label was created automatically when the Kubernetes clusters were deployed by VMware Enterprise PKS. 

 

Each Kubernetes worker node was placed into an availability zone. Then availability zones were defined in the VMware Enterprise PKS configuration tab of Pivotal Operations Manager. We introduced them in the “Create VMware Enterprise PKS Availability Zones” section. The value of “bosh.zone” could be “pks-comp-az1” or “pks-comp-az2” or “pks-comp-az3”.

 

The ‘podAntiAffinity’ rule will look at the “bosh.zone” lable and ensure the pods are not placed to worker nodes with the same label. In this way, we guaranteed that the pods in a stateful set were placed into different availability zones to ensure high availability.

 

Use the ‘kubectl get po -o wide’ to validate that the pods are indeed on different nodes in different availability zones.

Use Harbor for Local Image Registry

Harbor is integrated into VMware Enterprise PKS and we can use it for the local image registry. Harbor also resided on the vSAN datastore. In the previous example of the MongoDB’s ConfigDB yaml, we used the following line to indicate that we were using a local harbor registry instead of the Docker.com’s public registry:

 

image: harbor1.pks.lab/library/mongo:3.6.9

 

To achieve this, we need to store images into harbor before we deploy any Kubernetes workloads.

On a Linux client that can access Internet, run the following commands (using MongoDB v3.6.9 as example):

 

$ docker pull mongo:3.6.9

$ docker tag mongo:3.6.9 harbor1.pks.lab/library/mongo:3.6.9

$ docker push harbor1.pks.lab/library/mongo:3.6.9

 

We pulled the MongoDB image from the docker.com’s public registry and stored it to our local Linux client. Then we tagged it with our custom harbor domain and custom library. Finally, we pushed it to our local harbor registry.

 

After we pushed the image to local harbor registry, we can define our custom image URL in the stateful set’s yaml configuration file. The Kubernetes clusters can pull the images from the local harbor registry in case they did not have Internet access.

Delete Stateful Set

We used the following commands to delete the ‘ConfigDB’ stateful set:

 

$ kubectl delete -f mongodb-configdb-service.yml

Figure 17. Deletion of Stateful Set and Persistent Volume Claim

From Figure 17, we can see that after the stateful set was deleted, the corresponding persistent volume claims were reserved, which was the reason that they were called persistent volumes. VMware Enterprise PKS detached them from worker VMs and they were standalone VMDKs in the vSAN datastore.

If we create the stateful set later without changing the configuration, these reserved persistent volume claims would be reused, thus the data are persistent. This process is totally automatic. Users need to operate the ‘kubectl’ command.

If we want to delete the data and start a fresh installation of the MongoDB stateful set, we could delete the persistent volume claims as shown in the screenshot with the following command:

$ kubectl delete pvc –-all

Delete Kubernetes Clusters Deployed by VMware Enterprise PKS

If the Kubernetes clusters used in previous sections are no longer needed, we could delete the cluster by using the ‘pks’ command line tool. We used the following command to delete a cluster:

$ pks delete-cluster k8s-cluster-1

Figure 18. Deletion of a Kubernetes Cluster Deployed by VMware Enterprise PKS

During the deletion as shown in Figure 18, the ‘Last Action State’ was shown as ‘in progress’. After the deletion was successfully completed, we used ‘pks cluster k8s-cluster-1’ to query the status of the cluster and the cluster was not found.

 

The corresponding NSX-T load balancer, routers, and switches were automatically deleted by VMware Enterprise PKS. 

Monitoring and Resiliency Testing

Monitoring Kubernetes Clusters with vRealize Operations

The vRealize Operations Management Pack for Container Monitoring can be downloaded in this page. It also contains the detailed usage and configuration guide. 

In this reference architecture, we only highlighted some most useful dashboards and the reported alerts when we did the resiliency tests.

After we installed the “vRealize Operations Management Pack for Container Monitoring” in vRealize Operations Manager, we could configure our VMware Enterprise PKS adapter in the “Administration” tab as in Figure 19.

Figure 19. Configuration of PKS Adapter in vRealize Operations Manager

It is only necessary to configure the “PKS Adapter”. We do not need to configure any “Kubernetes Adapter” as they will be automatically configured by the VMware Enterprise PKS Adapter. 

In our case, we deployed two Kubernetes clusters called “k8s-cluster-1” and “k8s-cluster-2” as described in the previous sections. The monitoring adapters were automatically for these two clusters. 

After a while, we could monitor the Kubernetes cluster status as depicted in Figure 20. By clicking on various parts of the main dashboard, we could navigate into the details of each part such as nodes, pods, containers and so on.

Figure 20. Monitoring Dashboard of Kubernetes Clusters Deployed by VMware Enterprise PKS

Monitoring and Resiliency Testing

Monitoring Kubernetes Clusters with vRealize Operations

The vRealize Operations Management Pack for Container Monitoring can be downloaded in this page. It also contains the detailed usage and configuration guide. 

In this reference architecture we only highlighted some most useful dashboards and the reported alerts when we did the resiliency tests.

After we installed the “vRealize Operations Management Pack for Container Monitoring” in vRealize Operations manager, we could configure our VMware Enterprise PKS adapter in the “Administration” tab as in Figure 19.

Figure 19. Configuration of PKS Adapter in vRealize Operations Manager

It is only necessary to configure the “PKS Adapter”. We do not need to configure any “Kubernetes Adapter” as they will be automatically configured by the VMware Enterprise PKS Adapter. 

In our case, we deployed two Kubernetes clusters called “k8s-cluster-1” and “k8s-cluster-2” as described in the previous sections. The monitoring adapters were automatically for these two clusters. 

After a while, we could monitor the Kubernetes cluster status as depicted in Figure 20. By clicking on various parts of the main dashboard, we could navigate into the details of each part such as nodes, pods, containers and so on.

Figure 20. Monitoring Dashboard of Kubernetes Clusters Deployed by VMware Enterprise PKS

Monitoring NSX-T with vRealize Operations

The vRealize Operations Management Pack for NSX-T can be downloaded in this page. It also contains the detailed usage and configuration guide. 

In this reference architecture, we only highlighted some most useful dashboards and the reported alerts when we did the resiliency testings.

After we installed the “vRealize Operations Management Pack for NSX-T” in vRealize Operations Manager, we could configure the NSX-T adapter in the “Administration” tab as in Figure 21.

Figure 21. Configuration of NSX-T Adapter in vRealize Operations Manager

After a while, we could monitor the NSX-T cluster status as depicted in Figure 22. By clicking on various parts of the main dashboard, we could navigate into the details of each part such as controllers, logical switches, logical routers and so on.

Figure 22. Monitoring Dashboard of NSX-T

Analyzing NSX-T logs with vRealize Log Insight

The vRealize Log Insight Pack for NSX-T can be downloaded in this page. It also contains the detailed usage and configuration guide. 

In this reference architecture, we only highlighted some most useful dashboards and the reported alerts when we did the resiliency testing.

After we installed the “vRealize Log Insight for NSX-T” in vRealize Operations manager, we must set the syslog address in NSX-T Manager, NSX-T Controllers, NSX-T Edges and ESXi hosts to point to the vRealize Log Insight server address.

After a while, we could monitor the NSX-T logs and any potential errors as depicted in Figure 22. This testbed was working properly so no errors were found.

Figure 23. Dashboard of NSX-T in vRealize Log Insight

Kubernetes Primary Node Failure Testing

We simulated a Kubernetes primary node failure by issuing a “Power Off” command to one of the primary nodes virtual machine in vCenter. Then BOSH director deployed by Pivotal Operations Manager monitored that the virtual machine was not accessible. Therefore, BOSH deleted the original primary node virtual machine and created a new one. The tasks procedure was recorded as in Figure 24. The ‘Task Console’ had the most recent task on top of the screenshot and the oldest task at bottom of the screenshot.

Figure 24. Kubernetes Primary Node Failure Testing

We had 3 availability zones created by VMware Enterprise PKS and each primary node lied in each availability zone. During the reconfiguration of the failed primary node, the surviving 2 primary nodes kept working so service was not interrupted.

Kubernetes Worker Node Failure Testing

We simulated a Kubernetes worker node failure by issuing a “Power Off” command to one of the worker nodes virtual machine in vCenter. To get the impacted nodes and pods information, we used the following commands in Figure 25.

Figure 25. Impacted Nodes and Pods in Kubernetes Worker Node Failure Testing

We powered off the worker node with IP address of 172.15.1.7 and we could verify that the pod ‘mongod-configdb-2’ was impacted.

Similar to the primary node failure testing, BOSH director deployed by Pivotal Operations Manager monitored that the virtual machine was not accessible. Therefore, BOSH deleted the original primary node virtual machine and created a new one. The Kubernetes monitoring in vRealize Operations also detected that a node was down as in Figure 26. The number of nodes went down from 3 to 2 and the node was not shown in the healthy nodes widget.

Figure 26. Dashboard of Kubernetes Monitoring during a Worker Node Failure

After BOSH created the new worker node, the dashboard showed the number of nodes went back to 3 and the pod ‘mongod-configdb-2’ was re-created on the new worker node.

Figure 27. Dashboard of Kubernetes Monitoring after the Worker Node Recovered

We had 3 availability zones created by VMware Enterprise PKS and each worker node lied in each availability zone. The pods were controlled by a StatefulSet and they were forced to reside on different worker nodes in different availability zones by the ‘PodAntiAffinity’ setting. During the reconfiguration of the failed worker node, the surviving 2 pods of the StatefulSet on the surviving 2 worker nodes kept working so service was not interrupted.

ESXi Host Failure Testing

We simulated an ESXi host failure by issuing a “power reset” command to one of hosts through the server’s BaseBoard Management Console. 

The vSAN cluster had 3 host groups and each host group contains 2 ESXi hosts. Unlike the case in a Kubernetes primary or worker node failure, an ESXi host failure would trigger the vSphere HA to restart the impacted virtual machines in the surviving host of the same host group of the failed host. 

The vRealize Operations monitoring showed the same as in the Kubernetes worker node failure case. One of the worker nodes went down due to the host failure and after vSphere HA restarted the virtual machines the dashboard went normal as in Figure 26 and Figure 27.

One of the differences between host failure and a worker node failure is that host failure would impact vSAN’s health temporarily. Once the host was powered off, vSAN Health page in vCenter would report some unhealthy items as in Figure 28.

Figure 28. vSAN Health Monitoring During a Temporary ESXi Host Failure

When the host’s power was resumed and after ESXi booted successfully, the vSAN Health monitoring items became all green in a short time.

We used an FTT=1 setting for the virtual machines in this vSAN cluster. With FTT=1, vSAN can tolerant one physical disk failure or one host failure from the storage perspective. vSAN can ensure the storage is accessed by virtual machines after the failure.

So one host failure would not cause any storage interruption. From VMware Enterprise PKS and Kubernetes clusters’ perspective, there was no impact from the storage layer. They only needed to wait for the nodes and pods getting restarted and then service was resumed.

Mixed Workloads Performance Testing

VMware Enterprise PKS Plan Definition

In VMware Enterprise PKS, a plan is used to define the VM’s CPU, memory and disk size of the primary nodes and worker nodes. We could define up to 10 different plans in VMware Enterprise PKS for running different workloads.

In this paper, we defined 3 plans as follows. Refer to the Pivotal document for primary and worker nodes sizing based on custom production workloads.

Table 1. VMware Enterprise PKS Plans definition in this reference architecture

Plan Name

Primary node

Worker Node

‘small’ plan

cpu: 1, ram: 2GB, disk: 8GB

cpu: 1, ram: 2GB, disk: 8GB

‘large’ plan

cpu: 2, ram: 8GB, disk: 32GB

cpu: 4, ram: 16GB, disk: 32GB

‘2xlarge’ plan

cpu: 2, ram: 8GB, disk: 32GB

cpu: 8, ram: 32GB, disk: 64GB

MongoDB and MySQLPerformance in VMware Enterprise PKS

MongoDB Testing Procedure

We referred to k8smongodb for deploying sharded MongoDB clusters.

We used the ‘pkscli’ command line tool to deploy Kubernetes clusters. Each Kubernetes cluster used for MongoDB was deployed with the ‘2xlarge’ plan described in the ‘PKS Plan’ section. Each Kubernetes cluster contained 3 primary node and 6 worker nodes.

For each MongoDB cluster, we deployed 1 ‘configdb statefulset’ to act as the internal configuration database of MongoDB. There were also 4 ‘shards statefulsets’ as MongoDB’s actual data nodes and 1 ‘mongos statefulset’ as the routing service of MongoDB.

We used YCSB and the workload type of ‘workload A (50% write/50% read)’ as described in the Test Tool and Mixed Workload section. 

Each round of test costed 30 minutes for warming up and 1 hour for performance testing. 

MySQL Testing Procedure

We referred to this Kubernetes tutorial for deploying MySQL primary-secondary clusters.

We used the ‘pkscli’ command line tool to deploy Kubernetes clusters. Each Kubernetes cluster used for MySQL was deployed with the ‘2xlarge’ plan described in the ‘PKS Plan’ section. Each Kubernetes cluster contained 1 primary node and 4 worker nodes.

For each MySQL cluster, we deployed 1 primary pod and 2 secondary pods. They are running in the primary-secondary mode.

We used SysBench to test the MySQL clusters performance as described in the Test Tool and Mixed Workload section.

Each round of test costed 30 minutes for warming up and 1 hour for performance testing. 

Note: The transaction time was measured by the SysBench client for a database transaction operation. It is not only the storage layer read and write latency. So the average transaction time was usually much higher than the storage backend read and write latency. It is a common performance index in a SQL database testing.

Mixed workloads performance result

We deployed 2 Kubernetes clusters for running 2 MongoDB clusters separately. Meanwhile, we also deployed 2 Kubernetes clusters for running 2 MySQL clusters. 

The 4 clusters are running simultaneously. We aggregate MongoDB results and MySQL results separately.

We used 2 YCSB clients and 2 SysBench clients simultaneously to test the performance with MongoDB and MySQL running simultaneously.

The results are shown in 

Table 2. Testing result of MongoDB Performance

MongoDB Results

Operations/Second

Write Latency (ms)

Read Latency (ms)

Value

20199

3.785

2.35

Best Practices

When configuring VMware Enterprise PKS in a vSAN cluster, consider the following best practices: 

  • Enable HA in the vSAN cluster.
  • Enable DRS in the vSAN cluster.
  • Use Anti-Affinity DRS rule to force the virtual machines of the NSX-T controller cluster reside on separate physical hosts.
  • Use Anti-Affinity DRS rule to force the virtual machines of the NSX-T edge cluster reside on separate physical hosts.
  • Enable the Multi-AZ configuration.
  • Use at least 3 primary nodes and 3 worker nodes in Kubernetes clusters to get high availability with Multi-AZ.
  • Use Kubernetes’ anti-affinity or affinity rules to pods to place them into different availability zones.
  • If Harbor’s data disk is smaller than 255GB, increase its stripe width to improve Harbor’s performance.
  • Set MTU equals to 9000 for all the physical switches.
  • Choose different VMware Enterprise PKS plans for different workload demands.

Conclusion

Overall, deploying, running, and managing VMware Enterprise PKS on VMware vSAN provides predictable performance and scalability by taking advantages of the security, performance, scalability, operational simplicity, and cost-effectiveness of vSAN. Furthermore, by combining VMware Enterprise PKS and vSAN as a solution, all storage managements including infrastructure and operations of vSphere and vSAN move into a single software stack, thus you do not need two separate infrastructures for traditional storages and containers. 

Reference

See more vSAN details and customer stories: 

About the Author

Victor Chen, Solutions Architect in the Product Enablement team of the Storage and Availability Business Unit wrote the original version of this paper. 

Myles Gray, Senior Technical Marketing Architect of the HCI Business Unit contributed to the paper contents. 

The follow colleagues also contributed to the VMware Cloud Foundation parts of the paper:

  • Jim Senicka, Director of technical marketing of the HCI Business Unit
  • Kevin Tebear, Staff Technical Marketing Architect of the HCI Business Unit
  • Kyle Gleed, Manager of technical marketing of the HCI Business Unit

Filter Tags

vSAN Reference Architecture