Red Hat OpenShift Container Platform 4.13 on VMware Cloud Foundation 5.1 with NVIDIA AI Enterprise

Executive Summary

Business Case

Red Hat® OpenShift® Container Platform (OCP) provides automated installation, upgrades, and lifecycle management throughout the container stack. The container stack includes the operating system, Kubernetes, cluster services, and applications on any cloud. OpenShift helps teams build with speed, agility, confidence, and choice. It is focused on security at every level of the container stack and throughout the application lifecycle. OpenShift is backed by one of the leading Kubernetes contributors and open-source software companies, providing long-term and enterprise support. OpenShift is a consistent hybrid cloud foundation for building and scaling containerized applications. It simplifies and accelerates the development, delivery, and life cycle management of a hybrid mix of applications, consistently anywhere across on-premises, public clouds, and edge. Red Hat OpenShift Container Platform delivers a single, consistent Kubernetes platform anywhere that Red Hat Enterprise Linux® runs. The platform ships with a user-friendly console to view and manage all users’ clusters so users have enhanced visibility across multiple deployments.

The manageability of operating an OpenShift environment with virtualized infrastructure can be improved over the management of traditional IT infrastructure on bare metal, since the demand for resources can fluctuate with business needs, leaving the OpenShift cluster either under-powered or over-provisioned. IT needs a more flexible, scalable, and secure infrastructure to handle the ever-changing demands of OpenShift. VMware Cloud Foundation™ is a single architecture that is easy to deploy and can provision compute, network, and storage on demand. It delivers flexible, consistent, secure infrastructure and operations across private and public clouds, making it ideally suited to meet the demands of OpenShift. VMware Cloud Foundation protects network and data with micro-segmentation and satisfies compliance requirements with data-at-rest encryption. Policy-based management delivers business-critical performance. VMware Cloud Foundation delivers a consistent hybrid cloud foundation for building and scaling containerized applications. It simplifies and accelerates the development, delivery, and life cycle management of a hybrid mix of applications, consistently anywhere across on-premises, public clouds, and edge. VMware Cloud Foundation delivers policy-based automation for workload domains, enabling IT teams to easily provision and manage OpenShift clusters. VMware Cloud Foundation is designed to maximize hardware usage, significantly saving costs. It also provides interoperability and open standards, ensuring compatibility with a broad range of on-premises and public cloud infrastructures.

Artificial Intelligence (AI) is the simulation of human intelligence in machines that are programmed to think and act like humans, and it involves the development of algorithms and computer programs that can perform tasks that typically require human intelligence such as visual perception, speech recognition, decision-making, and language translation. NVIDIA AI Enterprise (NVAIE) is an end-to-end, cloud-native software platform that accelerates data science pipelines and streamlines the development and deployment of production-grade AI applications, including generative AI. It is certified to deploy anywhere, from the enterprise data center to the public cloud, and includes global enterprise support and training.

In this solution, we provide the generic design and deployment guidelines for running the OpenShift Container Platform on VMware Cloud Foundation, with AI applications support powered by NVIDIA AI Enterprise.

Note: If users do not have the requirement for running AI applications, users can also follow this paper for architecting the OpenShift Container Platform on VMware Cloud Foundation. In this case, users can skip the sections about configuring and using NVAIE.

Why Run OpenShift Container Platform on VMware Cloud Foundation?

VMware Cloud Foundation combines automation with a standardized and repeatable approach to infrastructure, giving IT Operations the infrastructure agility necessary to support developers by providing developer-ready infrastructure.

The infrastructure automation capabilities of VMware Cloud Foundation enable administrators to quickly deploy, manage, and scale the underlying infrastructure with cloud-like agility and at the speed of business.

By running OpenShift on VMware Cloud Foundation, you get all the benefits of a modern private cloud based on the proven VMware Software-Defined Data Center architecture:

• A consistent and repeatable approach to standardized Infrastructure.
• The advanced automation eliminates human error and fosters admin productivity.
• Cloud scale and agility that enables easy scale at the speed of the business. 
• Integrate with the well-proved networking solution from VMware: VMware NSX™.
• Integrate with the well-proved storage solution from VMware: vSAN™.
• Leverage the advantages of VMware vSphere® features, such as VMware vSphere Distributed Resource Scheduler™ (DRS), vSphere vMotion®, Fault Tolerance, and so on.
• Integrate with NVIDIA AI Enterprise (NVAIE) and use vGPU for fine grained GPU resource usage.

OpenShift on vSphere Benefits over OpenShift on Physical Server 

vSphere offers efficiency and security in storage, networking, and memory usage. Meanwhile, virtualization brings operational savings to containerized workloads.

By virtualizing OpenShift, it supports many small OpenShift clusters in a vSphere cluster. This provides granular lifecycle management of Kubernetes, stronger isolation, and smaller blast zones.

Virtualization Brings Operational Savings to Containerized Workloads

With a well-integrated Kubernetes platform such as OpenShift or Tanzu, vSphere provides a developer-ready infrastructure for container platforms including OpenShift. Well-known vCenter tools and processes can now manage both traditional VM-based and containerized workloads across your hybrid cloud. vSphere brings trusted capabilities such as VMware vSphere High Availability (vSphere HA) and policy-based management to ensure availability and resiliency for all workloads. vSphere also enhances the security of containers by naturally providing isolation of the VMs. In addition, vSphere life-cycle management and enterprise resiliency reduce the administration time required to manage bare metal updates and failures. All the above benefits help improve daily operational efficiency for DevOps. 

To learn more about these benefits in detail, check out  Why Choose VMware Virtualization for Kubernetes and Containers.

Higher Container Pod Density Results in Lower Capex

OpenShift 4.x is limited to 500 pods per Worker Node. With many hardware configurations today, this results in underutilized hardware. By using virtualized worker nodes, the number of pods per physical server can easily exceed 500 and subsequently decrease Capex. The higher pod density translates to lower cost as the number of physical hosts required to run the same number of containers will be lower. By abstracting physical hardware, running OpenShift on vSphere allows for better utilization of resources than OpenShift on bare metal, which is a key advantage that virtualization offers over bare metal.

Audience

This reference architecture paper is intended for the following audiences:

• Corporate CTOs and CIOs who are architecting OpenShift or Kubernetes in a private data center or in hybrid-Cloud.
• vSphere VI administrators who are familiar with VMware virtualized infrastructure and need to deploy and manage OpenShift in a virtualized environment.
• DevOps who are deploying, managing, or using OpenShift on vSphere.
• Enterprises and users who want to run AI applications.
• Any other engineer/operator/end-user who are interested in OpenShift/Kubernetes/vSphere and have the basic knowledges about VMware Cloud Foundation, VSAN, NSX, Cloud Native Storage (CNS), Container Storage Interface (CSI), OpenShift, Kubernetes, and AI.

Technology Overview

Solution technology components are listed below:

• VMware Cloud Foundation 
• VMware vSphere
• VMware vSAN
• VMware NSX Data Center 
• Kubernetes vSphere CSI 
• Red Hat OpenShift Container Platform
• VMware Container Networking™ with Antrea™ for OpenShift
• NVIDIA AI Enterprise

VMware Cloud Foundation

VMware Cloud Foundation is an integrated software stack that combines compute virtualization (VMware vSphere), storage virtualization (VMware vSAN), network virtualization (VMware NSX), and cloud management and monitoring (VMware vRealize® Suite) into a single platform that can be deployed on-premises as a private cloud or run as a service within a public cloud. This documentation focuses on the private cloud use case. VMware Cloud Foundation bridges the traditional administrative silos in data centers, merging compute, storage, network provisioning, and cloud management to facilitate end-to-end support for application deployment. See Getting Started with VMware Cloud Foundation for details. 

VMware vSphere

VMware vSphere is VMware's virtualization platform, which transforms data centers into aggregated computing infrastructures that include CPU, storage, and networking resources. vSphere manages these infrastructures as a unified operating environment and provides operators with the tools to administer the data centers that participate in that environment. The two core components of vSphere are ESXi™ and vCenter Server®. ESXi is the hypervisor platform used to create and run virtualized workloads. vCenter Server is the management plane for the hosts and workloads running on the ESXi hosts.

VMware vSAN

VMware vSAN is the industry-leading software powering VMware’s software-defined storage and Hyperconverged Infrastructure (HCI) solution. vSAN helps customers evolve their data center without risk, control IT costs, and scale to tomorrow’s business needs. vSAN, native to the market-leading hypervisor, delivers flash-optimized, secure storage for all of your critical vSphere workloads, and is built on industry-standard x86 servers and components that help lower TCO in comparison to traditional storage. It provides the agility to scale IT easily and offers the industry’s first native HCI encryption.

VMware NSX Data Center

VMware NSX Data Center is the network virtualization and security platform that enables the virtual cloud network, a software-defined approach to networking that extends across data centers, clouds, and application frameworks. With NSX Data Center, networking and security are brought closer to the application wherever it is running, from virtual machines to containers to bare metal. Like the operational model of VMs, networks can be provisioned and managed independently of the underlying hardware. NSX Data Center reproduces the entire network model in software, enabling any network topology—from simple to complex multi-tier networks—to be created and provisioned in seconds. Users can create multiple virtual networks with diverse requirements, leveraging a combination of the services offered via NSX or from a broad ecosystem of third-party integrations ranging from next-generation firewalls to performance management solutions to build inherently more agile and secure environments. These services can then be extended to a variety of endpoints within and across clouds.

Red Hat OpenShift Container Platform

Red Hat OpenShift Container Platform ships with Red Hat Enterprise Linux® CoreOS for the Kubernetes control plane nodes and supports both Red Hat Enterprise Linux CoreOS and Red Hat Enterprise Linux for worker nodes. OpenShift supports the Open Container Initiative (OCI), which is an open governance structure around container formats and runtimes. OpenShift includes hundreds of fixes to defects, security, and performance issues for upstream Kubernetes in every release. It is tested with dozens of technologies and is a robust tightly integrated platform. OpenShift includes software-defined networking and validates additional common networking solutions. OpenShift also validates numerous storage and third-party plug-ins for every release.

See https://www.redhat.com/en/technologies/cloud-computing/openshift for detailed information regarding the OpenShift Container Platform.

VMware Container Networking with Antrea

VMware Container Networking with Antrea is a Kubernetes-native networking and security solution that provides network connectivity and security for pod workloads. It is designed to run anywhere Kubernetes runs, including on-premises, in the public cloud, and at the edge.

Antrea uses Open vSwitch as the networking data plane in every Kubernetes node. It simplifies Kubernetes networking by providing a unified networking stack across multiple managed Kubernetes providers, regardless of cloud platform. Antrea also offers centralized policy management and visibility with VMware NSX.

Antrea’s features include policy enforcement for managed Kubernetes services such as VMware Tanzu, AWS EKS, Azure AKS, Google GKE, OpenShift, Rancher, and more. It also provides micro-segmentation and IDPS for Kubernetes, enhancing security with advanced Antrea policies, protecting workloads with IDPS, and encrypting pod-to-pod traffic with IPSec.

Antrea’s policy tiering with RBAC allows grouping global policies into multiple-tiers and setting their evaluation order. It also allows only specific roles to control specific tiers. Antrea provides observability and troubleshooting by observing and diagnosing Antrea components and Traceflow and exporting network flows with IPFIX.

Regarding OpenShift, the Antrea Operator is responsible for deploying and managing the Antrea CNI plugin in an OpenShift cluster. The Antrea operator leverages a dedicated Custom Resource Definition (CRD) called AntreaInstall to store configuration parameters specific to both the antrea-controller and antrea-agent. The Antrea operator is now certified and available as a container network interface (CNI) plug-in and can be deployed starting from OpenShift 4.9.

See VMware Container Networking with Antrea Product Documentation for detailed information regarding Antrea.

NVIDIA AI Enterprise

NVIDIA AI Enterprise (NVAIE) is a software platform that helps businesses accelerate data science pipelines and streamline development and deployment of production-grade AI applications, including Generative AI. It is certified to deploy anywhere from the enterprise data center to the public cloud and includes global enterprise support and training. NVIDIA AI Enterprise is optimized for accelerated infrastructure, workload and infrastructure manageability, and workload portability across multi- and hybrid-cloud environments. It also provides regular security updates, assurance of API stability, and NVIDIA Enterprise Support. NVIDIA AI Enterprise is an ideal solution for enterprises that run their businesses on AI and rely on the security, support, and stability provided by NVIDIA AI Enterprise to ensure a smooth transition from pilot to production.

See NVIDIA AI Enterprise for detailed information regarding NVIDIA AI Enterprise.

Solution Configuration

This section introduces the following resources and configurations: 

• Architecture diagram
• Hardware resources 
• Software resources
• VMware Cloud Foundation Installation 
• Workload Domain Preparation 
• vSAN configuration
• Network configuration
• OpenShift installation with Antrea
• Integration of Antrea with NSX Data Center
• NVIDIA AI Enterprise Configuration

Architecture Diagram

In this solution, the VMware Cloud Foundation test environment was composed of a management domain and a workload domain. 

Typically, a VMware Cloud Foundation management domain can simultaneously manage multiple work domains. All the workload domains’ management VMs, such as vCenters and NSX Manager™, are placed in the management domain. All the workload domains can share one management domain.

Different workload domains can serve different business purposes. The one-to-many mapping simplifies the overall management of the whole VMware Cloud Foundation environment.

We deployed the OpenShift Container Platform in one of the workload domains. The workload domain also contains the VMware NSX Edge™ nodes. All other infrastructure VMs were in the shared management workload domain (figure 1).

This figure only shows the workload domain of OpenShift Container Platform that we focus on this architecture. 

The management domain can manage multiple work domains, but other work domains are not shown in this figure.

Figure 1. OpenShift on VMware Cloud Foundation Solution Standard Architecture

Notation in Figure 1:

• Temp Bootstrap: This is the temporary bootstrap node. It only exists during the installation process and will be deleted after the OpenShift Container Platform is fully deployed.
• Control plane-0,1,2: These are the control plane nodes of Kubernetes deployed and managed by OpenShift.
• Worker-0,1,2,3: These are the worker nodes of Kubernetes deployed and managed by OpenShift. We deployed 4 worker nodes as the starting point. More worker nodes can be added on demand through the OpenShift control plane.

The above architecture is called VMware Cloud Foundation’s standard architecture.

Note: Apart from the architecture in Figure 1, for a small test environment, the management domain and the workload domain can be consolidated into one domain. This architecture only requires a minimum of 4 ESXi hosts, which is ideal for a small or test environment. See VMware Cloud Foundation Architecture for details about VMware Cloud Foundation Consolidated Architecture Model.

Figure 2. OpenShift on VMware Cloud Foundation Solution Consolidated Architecture

In our solution, we tested both architectures: the VMware Cloud Foundation Standard Architecture and the VMware Cloud Foundation Consolidated Architecture.

Standard Architecture

For the standard architecture, we created a 4-node ESXi cluster for the VMware Cloud Foundation management domain, running management virtual machines and appliances. The management domain can be shared with other workload domains.

Table 1. Management Domain VMs

For the workload domain, we created another 4-node ESXi cluster with a separate NSX Fabric, deployed an NSX Edge Cluster, and deployed the OpenShift VMs in the workload domain. 

Table 2 shows the deployment of the workload domain edge nodes and OpenShift VMs. For the workload domain edge node, we recommend that NSX Edge transport nodes are deployed with “Large” form factor.

Table 2. Workload Domain VMs

Consolidated Architecture

For consolidated architecture, the virtual machines and their roles are the same. The difference is that in the consolidated architecture, there is only one cluster consisting of four ESXi hosts, and all virtual machines are in this cluster.

The above solution architecture as shown in figure 1 and figure 2, either in Standard Architecture Model or Consolidated Architecture Model, is called a building block for a basic installation of OpenShift with VMware Cloud Foundation. Based on the customer demands and database size, we can expand the workload domain to include more physical hosts. A cluster with vSAN enabled supports up to 64 physical hosts for non-stretched clusters. By adding more hosts to the vSAN cluster, not only the capacity of CPU and memory is increased for computing but also the capacity of vSAN storage is increased accordingly. This is one of the benefits of HCI that we can increase the capacity of computing and storage at the same time and proportionally.

Hardware Resources

In this solution, for the workload domain of OpenShift, we used a total of four Dell R7525 nodes. 

Each ESXi node in the cluster had the following configuration, as shown in Table 3. 

Table 3. Hardware Configuration for Workload Cluster

Software Resources

Table 4 shows the software resources used in this solution. 

Table 4. Software Resources

VMware Cloud Foundation Installation

The key steps for VMware Cloud Foundation installation are as follows:

1. Deploy a management domain.
2. Add ESXi hosts into the system.
3. Create a workload domain with the idle ESXi hosts.

Follow the Official Document for detailed information about VMware Cloud Foundation installation steps.

After the installation, just make sure that NSX and vSAN are successfully enabled on this workload domain. NSX and vSAN are integrated into this solution and will be used in the successive configurations of OpenShift.

Figure 3. Check the Configuration and State of VMware Cloud Foundation Deployment

Workload Domain Preparation

After the workload domain is created for OpenShift, NSX Managers are deployed in the Management Domain. However, the NSX Edge cluster is not deployed by default.

We must add an edge cluster after the workload domain is created, as shown in Figure 4.

After the edge cluster is deployed, the state should be ‘active’, and the edge nodes’ names are also shown as in Figure 4.

Figure 4. Enable NSX Edge Cluster on the Workload Domain

vSAN Configuration

The solution validation was based on a 4-node vSAN cluster as a building block. 

The validation tests were conducted using the default vSAN datastore storage policy of RAID 1 FTT=1, checksums enabled. The vSAN cluster has deduplication and compression deactivated, and no encryption. In the below sections, we explained the detailed configurations of the vSAN cluster and some items in the Storage Policy Based Management (SPBM).

Deduplication and Compression

The ‘Deduplication and Compression’ option was configured on the cluster level, and it can be enabled or deactivated for the whole vSAN cluster. While in our testing we deactivated it, by enabling it we can reduce the vSAN storage usage but induce higher latencies for the OpenShift application. This is a tradeoff for customers’ choices.

Failures to Tolerance (FTT)

Failures to Tolerance (FTT) is a configuration item in vSAN’s storage policy. For the ‘StorageClass’ in OpenShift and the corresponding vSAN’s storage policy, we recommended setting vSAN’s Failures to Tolerate (FTT) to 1. In our testing, we set FTT to 1 as the baseline. Do not set the FTT to 0 in an OpenShift with vSAN deployment because FTT=0 may possibly cause the data of the replications of the same pod to be stored in the same physical disk. This may cause data loss in case of a physical disk failure. 

Network Configuration

Figure 5 shows the VMware vSphere Distributed Switch™ network configuration for the OpenShift cluster in the workload domain of the VMware Cloud Foundation. NSX, which underlies the vSphere infrastructure, is used for the OpenShift cluster networking. To enable external access for the OpenShift cluster, an NSX Edge cluster is required to be deployed. Also, it is required to configure the BGP peering and route distribution of the upstream network. For more details, refer to VMware's NSX Document.

• The ‘VCF-xxxx-External-1’ and External-2 port groups are created by VMware Cloud Foundation automatically for NSX.
• The ‘xxxx-management’, ‘xxxx-vsan, ‘xxxx-vmotion’ are also created by VMware Cloud Foundation automatically. They are used for management, vSAN, and vMotion separately.
• The ‘ocp-segment’ is a logical switch manually created in the NSX. It is used for OpenShift VM nodes.

For the management domain, the port groups are similar. For each domain, two 100 GbE vmnics were used and configured with teaming policies. The management domain can be shared among different workloads.

Figure 5. vSphere Distributed Switch Network Configuration

Figure 6. NSX Overview Page

In the NSX Overview page, the ‘ocp-segment’ is used in this paper for reference. It is a logical segment in NSX. ‘ocp414-c2-segment’ is another segment for another OCP cluster. If more OCP clusters are required, more logical segments could be created accordingly. They are connected to the Tier-1 router, and the Tier-1 router is connected to the Tier-0 router.

The NSX controllers resided in the management domain. In our case, the OpenShift virtual machines were configured with a VM network called ‘ocp-segment’ on an NSX segment. DHCP is enabled on this segment, which is required by OpenShift. VMware vSphere vMotion®, vSAN, and VXLAN VTEP for NSX had another dedicated segment created. 

Jumbo Frame (MTU=9000) was enabled on the physical switches, vSAN VMkernel, and all the virtual switches to improve performance.

NSX Managers and Edges have more than one instance to form NSX clusters to achieve HA and better load balancing. Besides, based on workloads, the vCPU and memory may be adjusted to achieve better performance. Table 5 shows the configuration of the NSX Managers and edge nodes virtual machines. The NSX Managers reside in the management workload domain, so it will not cost the computing resources for OpenShift VMs. However, the NSX edge nodes reside in the OpenShift workload domain, and it will cost some CPU and memory resources. This should be taken into consideration while doing the sizing of the cluster before OpenShift is deployed.

Table 5. NSX Data Center VM Configuration

Some other configurations during the installation include the following key points:

DHCP

We must use DHCP in the vSphere cluster for the OpenShift cluster. Besides, we must make sure that the allocated IP addresses for virtual machines are persistent across rebooting or maintenance.

The DHCP server is used for the ‘VM Network’ of the control plane nodes and the worker nodes. In addition, the allocated network subnet must be the same as the API VIP and the Ingress VIP created below.

Since we used NSX in this reference architecture, turn on ‘DHCP’ service in the NSX segment for OpenShift. The DHCP server on NSX is HA enabled by default as it will fail over to the other Edge Node in case an Edge Node fails.

Alternatively, if there is already a DHCP server in the infrastructure layer outside of this VMware Cloud Foundation environment, the ‘DHCP Relay’ service can be configured in the logical segments in NSX and used for the OpenShift clusters.

Static IP Addresses

An installer-provisioned vSphere installation requires two static IP addresses:

• The API address is used to access the cluster API.
• The Ingress address is used for cluster ingress traffic.

These IP addresses should be in the same subnet as the DHCP range created above, which is also the ‘machineNetwork’ field in the OpenShift installer configuration file. This will be discussed in detail later in the ‘installation’ section.

DNS Server

We must have access to the DNS server and create two DNS records in the server for the API and Ingress IP addresses.

Table 6. DNS Server Components

See Installing a Cluster on vSphere for detailed information.

The static IP addresses and DNS entries are for OpenShift administrators and users to access the OpenShift (Kubernetes) cluster from an external network. It is not about the internal routing service or the internal name service inside an OpenShift (Kubernetes) cluster.

DHCP zones are used to define the DHCP server configuration for a specific subnet. A DHCP zone is a collection of DHCP options that are applied to a specific subnet. DNS zones are used to define the DNS server configuration for a specific domain. A DNS zone is a portion of the DNS namespace that is managed by a specific organization or administrator.

In this reference architecture, we recommend that for each OpenShift cluster set a dedicated DHCP zone and a dedicated DNS zone.

For example, if there are two OpenShift clusters named ‘ocp1’ and ‘ocp2’. The sample configurations could be:

Table 6. DHCP Zone, DNS Zone, and Subnets Example

DHCPv4 Option Code 15 is defined in [RFC2132] section 3.17, which specifies the domain name that the client should use when resolving host names by using the DNS. If your DHCP Server supports Option 15 for DHCP clients, we recommend that users enable this option to assign the domain name to DHCP clients (Openshift Control Plane nodes and Worker nodes). It would help all the nodes automatically register with DNS to save creating the entries. It also helps administrators manage the nodes easily with domain names but not IP addresses, especially for troubleshooting with many OpenShift clusters.

With VMware Cloud Foundation, CNS is already enabled by default. There is no need to manually install CNS or install any third-party CSI driver. The CSI driver is shipped with OCP by default with vSphere as the platform type, and it is used in this solution.

OpenShift Installation with Antrea

There are four methods to install OpenShift on vSphere: Agent-based Installer, Assisted Installer, User Provisioned Infrastructure’ (UPI), and Installer-Provisioned Infrastructure (IPI). IPI is an easier way for installation as it has more automation during deployment and follow-up management.

Selecting the installation method to use is dependent on the user demands and infrastructure requirements. An overview of these four methods is described in the documentation.

IPI is the only method that automatically creates the necessary resources (virtual machines and the template) on vSphere. As of OpenShift version 4.13, which is used in this reference architecture, IPI is fully supported for installing OpenShift on VMware Cloud Foundation. IPI is fully compatible with all the software components such as vSphere, Cloud Native Storage, and VMware Container Networking with Antrea.

During the installation process, we mainly used this OpenShift Document and VMware Container Networking with Antrea Document for reference.

We will demonstrate the major steps and highlights in this reference architecture while the above documents have more detailed information about the explanation of each step.

With IPI installation, OpenShift installer creates virtual machines automatically in the vSphere cluster. There is no need to manually create virtual machines.

For the network configuration, the major steps are:

1.In NSX Data Center, create the corresponding networking resources. With VMware Cloud Foundation, the edge cluster and overlay transport zones have already been created. We do not need to manually create them again.

 We need to manually create the following resources:

• A Tier-0 router

• A Tier-1 router

A segment: for example, named ‘ocp-segment’. This is a logical segment created in NSX. It is used for the primary ‘VM Network’ of control plane nodes, worker nodes, and the API virtual IP. If we plan to deploy more than one OCP cluster, we could create more segments to isolate the OCP clusters’ network.

2. Create a Linux virtual machine as the client, such as RHEL 9. We recommend connecting this client to the same segment as OpenShift nodes, ‘ocp-segment’ in our example. For this client, prepare the following basic software:

• Viewing and obtaining the pull secret from the Infrastructure Provider page. This page has the latest version of “oc” client and the "openshift-install" installer. If a specific version is required as in our case, go to the next step.
• Navigate to the OpenShift Container Platform downloads page on the Red Hat Customer Portal. Then download the OpenShift v4.13 Linux Client entry and save the file. This page has both the “oc” client and the "openshift-install" installer.
• Install a text editor such as Vim.
• Install an SSH client such as openssh-client.

3. On this Linux Client machine for OpenShift, create the install-config.yaml file.

The sample install-config.yaml file could be found on Github.

The command is:

./openshift-install create install-config –-dir ./

The required information is very simple as shown in Figure 7.

Figure 7. Create the install-config.yaml file for OpenShift

4.Pay attention to that in the OpenShift YAML file configuration above. Instead of the default ‘OVNKubernetes’ networking method, we should specify ‘antrea’ as the networking type like the following:

networking:

  clusterNetwork:

  - cidr: 10.64.0.0/14

hostPrefix: 23

  machineNetwork:

  - cidr: 172.16.62.0/26

  networkType: antrea

  serviceNetwork:

  - 192.168.64.0/18

The ‘machineNetwork’ field must be modified according to users’ environment. The machineNetwork needs to match the IP range of the ‘ocp-segment’ created earlier.

5.Run the following command to create manifests’ YAML files from the install-config.yaml file.

./openshift-install create manifests –dir ./

6.Go to https://downloads.vmware.com, search for “VMware Container Networking with Antrea” and select “VMware Antrea 1.8 Product Binaries”. Download the following:

• VMware Container Networking with Antrea (Advanced) – UBI image
• VMware Container Networking with Antrea, K8s Operator Image
• VMware Container Networking with Antrea, K8s Operator Manifests

Upload the images to an image registry that can be accessed by OpenShift control plane and worker nodes. The image registry could be DockerHub, Quay.io, users’ corporate private registry, or any other image registry you can access. 

If users are not familiar with the image management in a private image registry, at the time of writing these files are available on the VMware public repo; the paths are provided below for each image.

For example:

antreaImage: projects.registry.vmware.com/antreainterworking/antrea-ubi:v1.13.1_vmware.1

7.During step 6, the downloaded ‘K8s Operator Manifests’ file name is deploy.tar.gz. These manifests are for Antrea and they will be merged into the same directory of manifests created by openshift-installer. The manifests are version sensitive and they have to match the version of the image files that have been downloaded.

Extract the manifests files package:

$ tar xzvf deploy.tar.gz

./openshift-install create cluster –dir ./ --log-level=debug

export KUBECONFIG=/home/vmware/antrea-ocp/auth/kubeconfig

$ oc get nodes

Integration of Antrea with NSX Data Center

After the OpenShift cluster with Antrea is installed, we can integrate the cluster with NSX. This is also called the ‘Antrea-to-NSX Interworking’.

1.Create Principal Identity:

Follow the steps 1-6 in this NSX document to create Principal Identity and certifications automatically.

2.Change the operator configuration:

Follow the steps 2-8 in this Antrea document to install Antrea-to-NSX networking.

The Antrea cluster is now controlled by NSX. Then, the NSX administrators can see the network policies of Antrea.

NVIDIA AI Enterprise Configuration

This is an optional step. Users who do not intend to deploy AI applications can omit this section and starting using OpenShift Container Platform on VMware Cloud Foundation after the above installation processes are completed.

This section describes the procedures and usages of enabling and using NVIDIA AI Enterprise for AI applications with OpenShift Container Platform on VMware Cloud Foundation.

NVIDIA AI Enterprise Installation Steps

1.Install NVIDIA AI Enterprise ESXi driver.

2.Add vGPU PCI device:

1)Sequentially do the following operations on each of the OpenShift worker nodes.

2)For each OpenShift worker node, power on the virtual machine from vSphere client.

3)Edit the virtual machine settings.

Click ADD NEW DEVICE, select PCI Device.

In the Device Selection page, the devices starting with name ‘grid_100’ are vGPUs. The suffix 40c, 20c, and others are the profiles of the vGPU. Choose a proper sized vGPU based on AI workloads. We used grid_a100-40c in our example.

4)Add two parameters in the Advanced Parameters page:

   Add an entry with the attribute name ‘pciPassthru.use64bitMMIO’ with value ‘TRUE’.

   Add an entry with the attribute name ‘pciPassthru.64bitMMIOSizeGB’ with value ‘512’.

5)Save the changes and power on the OpenShift worker node virtual machine:

          Repeat this procedure for the rest of the OpenShift worker node virtual machines.

3.Install the ‘Node Feature Discovery’ OpenShift Operator:

Go to the OpenShift Web Console. On the OperatorHub page, search for the ‘Node Feature Discovery’ OpenShift Operator.

Click and install it with the default settings.

Then conduct the following steps:

1)Navigate to the Operators → Installed Operators page.

2)Find Node Feature Discovery and see a box under Provided APIs.

3)Click Create instance.

4)Edit the values of the NodeFeatureDiscovery CR.

5)Click Create.

4.Install NVIDIA GPU Operator:

Search for ‘NVIDIA GPU Operator’. Click and install it with the default settings.

5.Create the ‘licensing-config’ ConfigMap: In the OpenShift Web Console, select Workloads->ConfigMap->Create ConfigMap.

The ConfigMap’s name must be licensing-config.

Add two ‘Key-value’ pairs:

• The first key is called client_configuration_token.tok. The value can be retrieved from your corporate’s NVIDIA license account.
• The second key is called ‘gridd.conf’. The content ‘FeatureType=1’ means we use vGPU.

6.Create the ‘ngc-secret’ Secret: In the OpenShift Web Console, select Workloads->Secrets->Create->Image Pull Secret.

Fill the secret’s name ‘ngc-secret’. Username is ‘$oauthtoken’ and the password is your API Key from NVIDIA NGC account.

7.Create ‘ClusterPolicyObject’.

On the ClusterPolicyObject detailed configuration page, there are several items that must be set. Otherwise, the  ‘ClusterPolicyObject’ creation would fail.

Name; gpu-cluster-policy

In the NVIDIA GPU/vGPU Driver config, fill in ‘licensing-config’ that was created in the previous step. Then check the ‘nlsEnabled’ box.

For the repository, type in ‘nvcr.io/nvaie’.

We used 535.129.03 for the driver image tag version.

We used vgpu-guest-driver-4-1 as the driver image.

In the ‘Image pull secrets’ field, fill in the name ‘ngc-secret’ we just created in the previous step.

Note: For different OpenShift Container Platform versions, different vGPU driver images are used.

• 4.9 is nvcr.io/nvaie/vgpu-guest-driver-3-0:525.60.13-rhcos4.9
• 4.10 is nvcr.io/nvaie/vgpu-guest-driver-3-0:525.60.13-rhcos4.10
• 4.11 is nvcr.io/nvaie/vgpu-guest-driver-3-0:525.60.13-rhcos4.11

We used nvcr.io/nvaie/vgpu-guest-driver-4-1:535.129.03-rhcos4.13 for OpenShift 4.13.

We can search for usable versions by using the NVIDIA ‘ngc’ command line tool. For example:

Leave all the other fields as default and click Create.

NVIDIA AI Enterprise Configuration Validation

After the GPU Operator installation and configuration processes are completed, we can use the ‘oc’ to verify the system is in healthy state.

For example, use the following command to validate the pods in the ‘nvidia-gpu-operator’ namespace are in ‘running’ state.

Use the following command to verify that the OpenShift worker nodes are recognized to have 1 GPU.

Solution Validation

Application Sample

Redis

Redis is an in-memory data structure store, used as a distributed, in-memory key–value database, cache, and message broker, with optional durability. Redis supports different kinds of abstract data structures, such as strings, lists, maps, sets, sorted sets, HyperLogLogs, bitmaps, streams, and spatial indices.

For data persistent in Redis deployment with Kubernetes, a persistent volume claim needs to be created, Thus, it is a prerequisite to test the CNS usage.

We used Redis as one of the applications for OpenShift’s functional validation.

CUDA Samples

‘CUDA Samples’ is a collection of containerized CUDA samples for CUDA developers, which demonstrates features in CUDA Toolkit. The details can be found in the NGC Catalog and Github.

We used CUDA samples to validate the functionality of vGPU in this architecture.

Functional Testing

Application Deployment Validation

After the OpenShift Container Platform was installed, we deployed a sample Redis application that has a persistent volume claim (PVC). So, we can validate that OpenShift can successfully provision an application and create PVC through CNS backed by vSAN.

There are two ways to deploy the sample Redis application:

1. Deploy Redis Operator through OpenShift Web Console.
2. Deploy Redis through the command line tool with the YAML configuration files.

Deploying Redis Operator through the OpenShift Web Console

Key steps are:

1. Go to https://console-openshift-console.apps.ocp-antrea.xxx.com 

Log in with the credentials provided during the installation:

INFO Login to the console with user: “kubeadmin”, and password: “xxxx”

2. Look for the Redis operator:

3. After the operator is installed, edit the configuration files.

4. Validate that Redis is successfully deployed.

Deploy Redis through Command Line Tool with the YAML Configuration Files

Redis has an approved operator in OpenShift's OperatorHub. However, for other applications, if there is no available operator, we can also deploy an application through the YAML configuration files. We take this sample Redis deployment as an example.

The sample code used in this example can also be found in this Github page.

The key steps are:

1. Create namespace:

   $ kubectl create ns redis

2. Create a storage class that uses the vSAN's RAID5 storage policy.

Note: There is already a default storage policy after the OpenShift Container Platform cluster is set up successfully, in our case the name is called 'thin-csi'. The RAID5 storage policy is just a showcase to explain how to create a storage class in OCP with vSAN. We can define different policies in vSAN to meet different applications' requirements. Then, storage classes can be created here, which are backed by different storage policies in vSAN.

$ kubectl apply -f vsan-raid5-sc.yaml

A sample vsan-raid5-sc.yaml is:

apiVersion: storage.k8s.io/v1 
kind: StorageClass 
metadata: 
  name: vsan-raid5-sc 
  annotations: 
storageclass.kubernetes.io/is-default-class: “false” 
provisioner: csi.vsphere.vmware.com 
parameters: 
  storagePolicyName: “vSAN-raid5-policy” 
  datastore: vSANDatastore 
  diskformat: thin 

3. Create a persistent volume claim:

$ kubectl apply -f pvc.yaml

A sample pvc.yaml is:

apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: fileserver-claim 
spec: 
  accessModes: 
  - ReadWriteOnce 
  storageClassName: “vsan-raid5-sc” 
  volumeName: fileserver 
  resources: 
requests: 
      storage: 1G 

4. Deploy Redis:

$ kubectl apply -f redis.yaml

A sample Redis Pod configuration YAML file is:

apiVersion: v1 
kind: Pod 
metadata: 
  name: redis-leader 
  labels: 
app: redis 
spec: 
  containers: 
- name: leader 
      image: harbor-repo.vmware.com/dockerhub-proxy-cache/redis 
      env: 
        - name: LEADER 
          value: “true” 
      volumeMounts: 
        - mountPath: /mnt/fileserver 
          name: redis-pvc 
      ports: 
        - containerPort: 6379 
  volumes: 
- name: redis-pvc 
      persistentVolumeClaim: 
        claimName: fileserver-claim 
        readOnly: false 

After the PVC is successfully created, we can find the corresponding PV entry on the CNS monitoring page in vSphere client as shown in Figure 8.

Figure 8. PVC Validation in Cloud Native Storage Monitoring Page in vSphere Client

GPU Application Deployment Validation

We used the ‘vectorAdd’ app in the CUDA samples.

Open and edit a file: sample-vectoradd.yaml

apiVersion: v1 
kind: Pod 
metadata: 
  name: vector-add 
spec: 
  restartPolicy: OnFailure 
  containers: 
  - name: vector-add 
    image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8 
    resources: 
      limits: 
        nvidia.com/gpu: 1 

oc apply -f sample-vectoradd.yaml

Check the result:

Best Practices

vSAN Best Practices

• Enable Jumbo Frame on the physical switches. Use Jumbo Frames on the vSAN VMKernel and all virtual switches.
• Apply the vSAN default storage policy or with higher Failures to Tolerate (FTT) value depending on the business data protection needs.

vSphere Best Practices

• Enable vSphere HA in the cluster.

We recommend enabling vSphere HA for the workload domain cluster.

If vSphere HA is enabled, in case of a physical host failure and there are enough remaining resources to satisfy the resource reservation like having a spare host, vSphere can automatically power on the impacted virtual machines on the other surviving hosts.

In case of a physical host failure and if there are not enough remaining resources to satisfy the resource reservation, vSphere HA would not restart the impacted virtual machines, which is by design. Because forcing a virtual machine restart on a surviving host may cause resource contention and imbalanced performance among the OpenShift nodes. We suggest that the resource reservation should at least be set to all the control plane nodes.

• Enable vSphere DRS in the cluster.
• “Fully Automated” option is configured in the DRS automation configuration.
• Place the control plane nodes on three different physical hosts to accommodate one host failure. DRS Anti-Affinity rules should be applied to separate the VMs to different physical hosts.
• Leverage DRS to automatically place the worker node virtual machines.

For DRS Anti-Affinity rules, see the DRS Documentation.

• Disable vSphere storage DRS in the cluster.
• vSphere storage DRS is not supported for OpenShift Container Platform. So, disable vSphere storage DRS in the cluster.
• Use compute-only vMotion and do not use Storage vMotion.
• OpenShift Container Platform generally supports compute-only vMotion.
• Storage vMotion of the vSphere volumes used by pods is not supported

Other Recommendations

• Use the same server model for the physical hosts in the workload domain.
• Follow the guidelines from Installing a Cluster on vSphere with Network Customizationsfor the detailed deployment and optimization items.
• Follow the VMware Container Networking with Antrea Document for network configuration. 

Conclusion

VMware Cloud Foundation delivers flexible, consistent, secure infrastructure and operations across private and public clouds. It is ideally suited to meet the demands of modern applications running on the Red Hat OpenShift Container Platform in a virtualized environment.

With VMware Cloud Foundation, we can easily manage the lifecycle of the hybrid cloud environment. Besides, we have a unified management plane for all applications including OpenShift. With VMware Cloud Foundation, we can leverage the leading virtualization technologies including vSphere, NSX, and vSAN.

In this solution paper, we demonstrated the architecture of running Red Hat OpenShift Container Platform with VMware Cloud Foundation. We showed the configuration details, the hardware resources, and the software resources used in the solution validation. We showed the various configuration options in addition to the best practices. VMware Cloud Foundation Manager provided lifecycle management. vSAN provided reliable, high-performance, and flexible storage to OpenShift. NSX Data Center provided the fine-grained, secured, and high-performance virtual networking infrastructure to OpenShift. Also, vSphere DRS and vSphere HA provided efficient resource usage and vSphere HA. All the above led to a consolidated solution of running the Red Hat OpenShift Container Platform with VMware Cloud Foundation.

References

• VMware Cloud Foundation
  • VMware Cloud Foundation 5.1 Release Notes
  • Get the Facts of VMware Cloud Foundation – Part 6
• VMware vSphere
• VMware vSAN
• VMware NSX Data Center
• Red Hat OpenShift Container Platform
• Solution YAML Files on Github

About the Author

Victor (Shi) Chen, Sr. Technical Marketing Manager of the Workload Technical Marketing team, wrote the original version of this paper.

The following reviewers also contributed to the paper contents:

• Mike Francis, Staff Customer Success Architect in VMware by Broadcom
• Chen Wei, Director of the Workload Technical Marketing team in VMware by Broadcom
• Catherine Xu, Senior Manager of the Workload Technical Marketing team in VMware by Broadcom
• Vivien Wang, Engineering Partner Manager in Red Hat
• Ramon Acedo Rodriguez, Product Manager in Red Hat