Using Argo CD for Continuous Delivery on VMware Cloud Foundation with Tanzu

Executive Summary

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. Argo CD can automatically control the cloud native applications’ deployment and manage their lifecycle. In addition, Argo CD is auditable, easy to understand, and easy to use.

Kubernetes is an orchestration platform that helps orchestrate containerized applications to run on a cluster of hosts. It is a system that automates the deployment and management of containerized applications on a given cloud platform or on-premises infrastructure.

VMware Cloud Foundation^TM with VMware Tanzu^TM is the best way to run Kubernetes workloads at scale. VMware Cloud Foundation is the hybrid cloud platform for managing virtual machines and orchestrating containers, built on full-stack hyperconverged infrastructure (HCI) technology.

With a single architecture that is easy to deploy, VMware Cloud Foundation can provision compute, network, and storage on demand. VMware Cloud Foundation protects network and data with micro-segmentation and satisfies compliance requirements with data-at-rest encryption. Policy-based management delivers business-critical performance. VMware Cloud Foundation delivers flexible, consistent, secure infrastructure and operations across private and public clouds and is ideally suited to meet the demands of Kubernetes.

Dell VxRail is the only fully integrated, preconfigured, and tested HCI system optimized for VMware vSAN™ and is the standard for transforming VMware environments and accelerating organizations’ adoption of DevOps and infrastructure as code operations. VxRail simplifies the complete HCI infrastructure lifecycle by providing advanced infrastructure automation capabilities that make it easier for customers to further simplify operations that are needed to adopt modern application platforms from day one through day two and beyond. VMware Cloud Foundation with Tanzu on VxRail helps make developing your cloud native strategy easy by leveraging consistent infrastructure and operations to support faster application development, scalability, and lifecycle management to ensure you are using the latest Kubernetes tools and features.

By running Argo CD on VMware Cloud Foundation with Tanzu, users can achieve the following benefits:

Providing developer-ready infrastructure, aligning DevOps and IT teams, and simplifying cloud operations.
Unified workload management for both virtual machines based, and Kubernetes based.
Unified management for compute, storage, and networking on the proven vSphere platform.
Continuous delivery of applications for both development and production environment.

This solution provides the generic design and deployment guidelines for Argo CD on VMware Cloud Foundation with Tanzu on VxRail.

Technology Overview

Solution technology components are listed below:

VMware Cloud Foundation with Tanzu

VMware vSphere® with Tanzu
VMware vSAN
VMware NSX® Data Center

Dell VxRail

Argo CD

VMware Cloud Foundation with Tanzu

VMware Cloud Foundation is an integrated software stack that combines compute virtualization (VMware vSphere), storage virtualization (VMware vSAN), network virtualization (VMware NSX), and cloud management and monitoring (VMware vRealize® Suite) into a single platform that can be deployed on-premises as a private cloud or run as a service within a public cloud. This documentation focuses on the private cloud use case. VMware Cloud Foundation bridges the traditional administrative silos in data centers, merging compute, storage, network provisioning, and cloud management to facilitate end-to-end support for application deployment.

VMware Cloud Foundation with Tanzu provides VMware Cloud Foundation functionalities as well as VMware Tanzu portfolio, which is the best way to run Kubernetes workloads at scale.

VMware vSAN

VMware vSAN is the industry-leading software powering VMware’s software defined storage and Hyperconverged Infrastructure (HCI) solution. vSAN helps customers evolve their data center without risk, control IT costs, and scale to tomorrow’s business needs. vSAN, native to the market-leading hypervisor, delivers flash-optimized, secure storage for all of your critical vSphere workloads and is built on industry-standard x86 servers and components that help lower TCO in comparison to traditional storage. It delivers the agility to scale IT easily and offers the industry’s first native HCI encryption.

vSAN simplifies Day 1 and Day 2 operations, and customers can quickly deploy and extend cloud infrastructure and minimize maintenance disruptions. vSAN helps modernize hyperconverged infrastructure by providing administrators a unified storage control plane for both block and file protocols and provides significant enhancements that make it a great solution for traditional virtual machines and cloud-native applications. vSAN helps reduce the complexity of monitoring and maintaining infrastructure and enables administrators to rapidly provision a file share in a single workflow for Kubernetes-orchestrated cloud native applications.

See VMware vSAN doc and VMware vSAN 7.0 Update 3 Release Notes for more information.

VMware NSX Data Center

VMware NSX Data Center is the network virtualization and security platform that enables the virtual cloud network, a software-defined approach to networking that extends across data centers, clouds, and application frameworks. With NSX Data Center, networking and security are brought closer to the application wherever it is running, from virtual machines to containers to bare metal. Like the operational model of VMs, networks can be provisioned and managed independently of the underlying hardware. NSX Data Center reproduces the entire network model in software, enabling any network topology—from simple to complex multitier networks—to be created and provisioned in seconds. Users can create multiple virtual networks with diverse requirements, leveraging a combination of the services offered via NSX or from a broad ecosystem of third-party integrations ranging from next-generation firewalls to performance management solutions to build inherently more agile and secure environments. These services can then be extended to a variety of endpoints within and across clouds.

VMware Tanzu Kubernetes Grid

VMware Tanzu Kubernetes Grid provides organizations with a consistent, upstream-compatible, regional Kubernetes substrate that is ready for end-user workloads and ecosystem integrations. You can deploy Tanzu Kubernetes Grid across software-defined datacenters (SDDC) and public cloud environments, including vSphere, Microsoft Azure, and Amazon EC2.

Tanzu Kubernetes Grid provides the services such as networking, authentication, ingress control, and logging that a production Kubernetes environment requires. It can simplify operations of large-scale, multi-cluster Kubernetes environments, and keep your workloads properly isolated. It also automates lifecycle management to reduce your risk and shift your focus to more strategic work.

VMware NSX-T Container Plug-in for Kubernetes

VMware NSX Container Plugin (NCP) provides the integration between VMware NSX-T™ Data Center and container orchestrators such as Kubernetes.

The main component of NSX Container Plugin runs in a container and communicates with NSX Manager and VMware Cloud Foundation with Tanzu. NSX Container Plugin monitors changes to containers and other resources and manages networking resources such as logical ports, switches, routers, and security groups for the containers by calling the NSX-T Policy API.

The NSX CNI plug-in monitors container life cycle events, connects a container interface to the guest vSwitch, and programs the guest vSwitch to tag and forward container traffic between the container interfaces and the vNIC.

Kubernetes vSphere CSI Driver

Cloud Native Storage (CNS) is a vSphere and Kubernetes (K8s) feature that makes K8s aware of how to provision storage on vSphere on-demand, in a fully automated, scalable fashion as well as providing visibility for the administrator into container volumes through the CNS User Interface within vCenter. Run, monitor, and manage containers and virtual machines on the same platform—in the same way:

Simplify your infrastructure needs, lifecycle, and operations.
Lower costs, using a platform you already know for consistent operations across workloads and across clouds.
Spend less time managing infrastructure and more time building apps that provide business value.

The main goal of CNS is to make vSphere and vSphere storage, including vSAN, a platform to run stateful Kubernetes workloads. vSphere has a great data path that is highly reliable, highly performant, and mature for enterprise use. CNS enables access of this data path to Kubernetes and brings an understanding of Kubernetes volume and pod abstractions to vSphere. CNS was first released in vSphere 6.7 Update 3.

Argo CD

Argo CD is a declarative and GitOps continuous delivery tool for Kubernetes.

Argo CD follows the GitOps pattern of using Git repositories as the source of truth for defining the desired application state. Kubernetes manifests can be specified in several ways:

kustomize applications
helm charts
ksonnet applications
jsonnet files
Plain directory of YAML/json manifests
Any custom config management tool configured as a config management plugin

See https://argo-cd.readthedocs.io/en/stable/ for detailed information regarding Argo CD.

Dell VxRail

The only fully integrated, pre-configured, and pre-tested VMware hyperconverged integrated system optimized for VMware vSAN and VMware Cloud Foundation™, VxRail provides a simple, cost effective hyperconverged solution that solves a wide range of operational and environmental challenges and supports almost any use case, including tier-one applications, cloud native and mixed workloads. Powered by next generation Dell PowerEdge server platforms and VxRail HCI System Software, VxRail features next-generation technology to future proof your infrastructure and enables deep integration across the VMware ecosystem. The advanced VMware hybrid cloud integration and automation simplifies the deployment of secure VxRail cloud infrastructure.

Test Tools

We leveraged the following monitoring and benchmark tools in this solution.

Monitoring Tools

Prometheus

Prometheus is an opensource systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community.

Prometheus collects and stores its metrics as time series data; for example, metrics information is stored with the timestamp at which time it was recorded, alongside optional key-value pairs called labels.

Grafana

Grafana is an open source visualization and analytics platform that unifies data sets across your company into an interactive diagnostic workspace. Grafana is built on a plug-in architecture that allows you to interact with the underlying data sources without creating data copies.

vSAN Performance Service

vSAN Performance Service is used to monitor the performance of the vSAN environment through the vSphere Client. The performance service collects and analyzes performance statistics and displays the data in a graphical format. You can use the performance charts to manage your workload and determine the root cause of the problems.

vSAN Health Check

vSAN Health Check delivers a simplified troubleshooting and monitoring experience of all things related to vSAN. Through the vSphere client, it offers multiple health checks specifically for vSAN including cluster, hardware compatibility, data, limits, and physical disks. It is used to check the vSAN health before the mixed-workload environment deployment.

This is only for vSAN health check. Customers can also enable VxRail cluster health monitoring for overall health monitoring.

Solution Configuration

This section introduces the resources and configurations:

Architecture diagram
Configuring a supervisor cluster
Provisioning Tanzu Kubernetes cluster
Argo CD installation
VMware Tanzu Kubernetes Grid Extensions installation
Application deployment by Argo CD
Hardware resources
Software resources
Network configuration

Architecture Diagram

The VMware Cloud Foundation test environment was composed of at least a management domain and a workload domain for showcase. We deployed Argo CD in a Tanzu Kubernetes Cluster in workload domain 1, and all other infrastructure VMs were in the separate management workload domain (Figure 1).

Figure 1. Basic Argo CD on VMware Cloud Foundation with Tanzu Solution Architecture

The applications deployed by Argo CD can reside in the same cluster as the Argo CD management components or be deployed into other Tanzu Kubernetes Clusters. In the figure above, they are shown as Tanzu Kubernetes Clusters 1 to N. For example, we can differentiate them as ‘development’ or ‘production’.

Figure 1 is a building block for running Argo CD on VMware Cloud Foundation with Tanzu. We can add more vSphere clusters into workload domain 1 to horizontally expand resources in the environment.

Besides, we can also horizontally expand resource by adding cluster to additional VMware Cloud Foundation workload domains as shown in Figure 2. Argo CD can connect to other Tanzu Kubernetes Clusters in other workload domain and deploy applications onto them.

Figure 2. Argo CD on VMware Cloud Foundation with Tanzu on More Workload Domains

In our solution, we created a 4-node VxRail P570F cluster for the VMware Cloud Foundation management domain, running management virtual machines and appliances. The management domain can be shared with other workload domains.

Table 1. Management Domain VMs

VM Role	vCPU	Memory (GB)	VM Count
Management Domain vCenter Server	4	16	1
SDDC Manager	4	16	1
Management Domain NSX-T Manager	6	24	3
Workload Domain NSX-T Manager	12	48	3
Workload Domain vCenter Server	8	28	1
VxRail Manager Appliance	2	8	1

For the workload domain, we created another 4-node VxRail P570F cluster with a separate NSX-T Fabric, deployed an NSX Edge Cluster, and deployed the vSphere with Tanzu supervisor cluster and Tanzu Kubernetes Clusters.

Table 2 shows the deployment of the workload domain edge nodes’ configurations. For the workload domain edge node, we recommend that NSX Edge transport nodes are deployed with the “Large” form factor.

Table 2. Workload Domain VMs

VM Role	Minimum vCPU	Minimum Memory (GB)	Storage	Deployment Size	VM Count
Workload Domain Edge node	8	32	200 GB	Large	2

Based on the customer demands and database size requirements, we can expand each workload domain to include more physical hosts. A cluster with vSAN enabled supports up to 64 physical hosts for non-stretched cluster. By adding more hosts to the vSAN cluster, not only is the capacity of CPU and memory increased for computing, but the capacity of vSAN storage is also increased accordingly. This is one of the benefits of HCI that we can increase the capacity of computing and storage at the same time and proportionally.

Configuring a Supervisor Cluster

After VxRail manager bootstrapped the whole VMware Cloud Foundation environment, NSX-T Data Center Networking and vSAN are configured and in place. When configuring the supervisor cluster, we can easily leverage NSX-T networking because it is an integral part of VMware Cloud Foundation.

See this document for the detailed configuration steps: Enable Workload Management with NSX-T Data Center Networking

After enabling the supervisor cluster, following the rest parts of the documentation to configure the ‘Content Library’ and download the corresponding command line tools.

Provisioning Tanzu Kubernetes Cluster

After the supervisor cluster is enabled, we can deploy at least one Tanzu Kubernetes Cluster for Argo CD workloads to run, as shown in the architecture figure.

Refer this document for the detailed configuration steps: Provisioning and Operating TKGS Clusters.

Make sure to meet the prerequisites before provisioning the Tanzu Kubernetes Cluster.

The ‘Tanzu Kubernetes Grid Service API’ originally came with the ‘v1alpha1’ version. It is recently upgraded to the ‘v1alpha2’ version. Different ‘Tanzu Kubernetes Grid Service API’ versions support different VMware Cloud Foundation with Tanzu version. Check your VMware Cloud Foundation version to use the correct API version for deployment.

Argo CD Installation

There are two ways to install Argo CD, First is to use the Argo CD operator and the second is to use bare YAML files.

Argo CD Operator

The latest version of Argo CD Operator is v0.2.0. Use with caution.

The Argo CD Operator’s Github page is: https://github.com/argoproj-labs/argocd-operator

Argo CD has its own official tutorial about how to install and use argocd-operator. We will not repeat it in this paper.

See https://argocd-operator.readthedocs.io/en/latest/install/start/ for installation guide.

See https://argocd-operator.readthedocs.io/en/latest/usage/basics/ for usage guide.

Bare YAML File Installation

In this solution, we also used Argo CD’s official installation guide for the deployment. This is based on YAML files.

HA stands for High Availability. In the above official guide, it introduces a non-HA deployment method. However, we recommend deploying the HA version of Argo CD. The only thing that needs to be modified during installation is replacing the manifest file from the default install.yaml to the HA version in the directory: ha/install.yaml. All the manifests of various deployment types are stored in this github repo: https://github.com/argoproj/argo-cd/tree/master/manifests

Tanzu Kubernetes Grid Service provisions Tanzu Kubernetes clusters with the PodSecurityPolicy Admission Controller enabled.

To run a privileged set of workloads in the Argo CD namespace, run the following command to create a ‘rolebinding’ to grant access to the service accounts in Argo CD namespace.

$ kubectl create rolebinding rolebinding-default-privileged-sa-ns_argocd --namespace=argocd --clusterrole=psp:vmware-system-privileged --group=system:serviceaccounts

We need to modify the ha-install.yaml file for Argo CD to expose its monitoring metrics to Prometheus.

For example, for the ‘argocd-metrics’ service, add three lines in the annotation section:

---

apiVersion: v1

kind: Service

metadata:

labels:

app.kubernetes.io/component: metrics

app.kubernetes.io/name: argocd-metrics

app.kubernetes.io/part-of: argocd

name: argocd-metrics

annotations:

prometheus.io/port: "8082"

prometheus.io/scrape: "true"

spec:

ports:

- name: metrics

port: 8082

protocol: TCP

targetPort: 8082

selector:

app.kubernetes.io/name: argocd-application-controller

---

Similarly, Add the annotation for ‘argocd-repo-server’ and ‘argocd-server-metrics’. Modify the ‘port’ section accordingly.

Then use the following command to deploy Argo CD.

Kubectl -n argocd apply -f ha-install.yaml

After Argo CD is successfully installed, we can see the following pods are created in the argocd namespace in the management cluster.

vmware@ubuntudt:~$ kubectl -n argocd get pods |grep argo

argocd-application-controller-0 1/1 Running 0 3d2h

argocd-dex-server-585d487b7f-4d7sk 1/1 Running 1 3d2h

argocd-redis-ha-haproxy-7fff9bc8d4-65qnf 1/1 Running 0 3d2h

argocd-redis-ha-haproxy-7fff9bc8d4-7wcb6 1/1 Running 0 3d2h

argocd-redis-ha-haproxy-7fff9bc8d4-pgs7f 1/1 Running 0 3d2h

argocd-redis-ha-server-0 2/2 Running 0 3d2h

argocd-redis-ha-server-1 2/2 Running 0 3d2h

argocd-redis-ha-server-2 2/2 Running 0 3d2h

argocd-repo-server-69779c7599-rfvzp 1/1 Running 0 3d2h

argocd-repo-server-69779c7599-sgkmx 1/1 Running 0 3d2h

argocd-server-8b499f766-dt9zz 1/1 Running 0 3d2h

argocd-server-8b499f766-gpzfb 1/1 Running 0 3d2h

The default services are:

vmware@ubuntudt:~$ kubectl -n argocd get svc |grep argo

argocd-dex-server ClusterIP 10.104.212.44 <none> 5556/TCP,5557/TCP,5558/TCP 3d2h

argocd-metrics ClusterIP 10.97.86.2 <none> 8082/TCP 3d2h

argocd-redis-ha ClusterIP None <none> 6379/TCP,26379/TCP 3d2h

argocd-redis-ha-announce-0 ClusterIP 10.107.69.225 <none> 6379/TCP,26379/TCP 3d2h

argocd-redis-ha-announce-1 ClusterIP 10.97.121.70 <none> 6379/TCP,26379/TCP 3d2h

argocd-redis-ha-announce-2 ClusterIP 10.100.103.72 <none> 6379/TCP,26379/TCP 3d2h

argocd-redis-ha-haproxy ClusterIP 10.96.48.144 <none> 6379/TCP 3d2h

argocd-repo-server ClusterIP 10.109.109.132 <none> 8081/TCP,8084/TCP 3d2h

argocd-server LoadBalancer 10.99.200.85 10.156.184.103 80:32458/TCP,443:31311/TCP 3d2h

argocd-server-metrics ClusterIP 10.102.212.177 <none> 8083/TCP 3d2h

VMware Tanzu Kubernetes Grid (TKG) Extensions Installation

We followed the Deploy TKG Extensions on Tanzu Kubernetes Clusters document to deploy the extensions.

Contour is a Kubernetes ingress controller that uses the Envoy reverse proxy. Deploy the TKG Extension for Contour Ingress to expose ingress routes to services running on Tanzu Kubernetes clusters. This extension is a prerequisite for Prometheus and Grafana.

Then we deployed the TKG extension of Prometheus, which is a system and service monitoring system. It is used to monitor the status of Argo CD itself as well as the applications managed by Argo CD.

The DNS domain name, user name, and password of Prometheus were configured during the installation process. If Prometheus extension is successfully deployed, we can see the ‘alerts’ page to check whether there are any initial alerts that reflects some misconfiguration in the environment.

In addition, during the Argo CD installation, we have configured Argo CD to expose its monitoring metrics to Prometheus. So Prometheus would get the statistics automatically. We can also check that in Prometheus->Status->Targets tab, Argo CD metrics services were successfully discovered.

Figure 3. Argo CD Metrics Services were Discovered

Figure 4. The Default ‘alerts’ Page to Show that the Prometheus Extension is Successfully Installed

We also deployed another TKG extension, Grafana, which lets you query, visualize, turn alerts on, and explore metrics no matter where they are stored. In addition, Grafana provides tools to form graphs and visualizations from the application data. Deploy the TKG Extension for Grafana to generate and view metrics for Tanzu Kubernetes clusters.

The DNS domain name, user name of password of Grafana are configured during the installation process. If Grafana extension is successfully deployed, we can open a web browser and use them to access Grafana. The Grafana default home page is shown in Figure 5. This page validated that Grafana was successfully installed.

Figure 5. The Default Home Page to Show that the Grafana Extension is Successfully Installed

After Grafana was installed, we could install Argo CD’s dashboard for Grafana to show a graphic UI to monitor Argo CD.

The dashboard source code can be found here: https://grafana.com/grafana/dashboards/14584

In the left panel of Grafana, click ‘+’ icon and choose ‘Import’ and then type the URL of the dashboard source.

Figure 6. Import the Argo CD Dashboard for Grafana

Optionally, choose a customized name or just leave it as the default ‘ArgoCD’.

Figure 7． Configure the Argo CD Dashboard for Grafana

After importing the dashboard, we can check that it is successfully installed as shown in Figure 8. This was the initial state so there was no data in it now. The monitoring source of the metrics came from Prometheus. Grafana can digest the monitoring status from Prometheus after some applications were deployed by Argo CD.

Figure 8. Initial State of the Argo CD Dashboard after Being Imported to Grafana

Finally, we also recommend deploying Harbor, an enterprise private image registry. By deploying harbor, we can use it for Tanzu Kubernetes Clusters to pull container images. The container images used in the Argo CD deployment and applications deployment are stored in harbor. So, Tanzu Kubernetes Clusters do not need to pull public images from the internet, such as docker.io.

Application Deployment by Argo CD

Argo CD supports continuously delivering application hosting in a Git or Helm repository. The Argo CD official document shows two examples: guestbook and helm-guestbook. They are the simplest applications to show the different configuration settings of Git or Helm based.

Firstly, we need to create a project in Argo CD if there were none default projects created as shown in Figure 9.

Figure 9. The Project Creation Page of Argo CD UI

After a project was created, we could set the parameters of the project, Define the desired repository, desired destination cluster, and so on. A wildcard ‘*’ means ‘any’ so there was no restriction. Use wildcards with caution.

Figure 10. The Project Configuration Page in Argo CD UI

After a project was configured, we could deploy an application. For example, the ‘guestbook’ from Argo CD’s official tutorial in Git format as show in Figure 11.

Figure 11. Creating the ‘guestbook’ Application to Validate Argo CD Functionality

After creation, the initial state showed that the application was in ‘missing’ and ‘OutOfSnc’ state. Wait for a while so the application could be synchronized.

Figure 12. Initial State of the Application Before It Was Synchronized

For validation purpose, we also deployed one of the most popular applications held in https://artifacthub.io/. We choose redis because it is popular and it has persistent volumes to show VMware CNS functionalities.

The difference from the ‘guestbook’ example above is that we must choose ‘HELM’ format in the ‘Repository URL’ section. Besides, we must manually specify a version, such as 15.5.0.

Figure. 13 Creating a ‘HELM’ Format Application ‘redis-dev’

We named the redis application ‘redis-dev’ to reflect that it is in active development, and we expect an update with committing new codes.

When code changes are committed and a new version of the application is deployed, we have various options like setting a target and choosing different sync options. The management and operations of Argo CD itself is beyond this paper’s scope. See Argo CD User Guide for details.

Figure 14 shows the dashboard of Argo CD after deploying some applications.

Figure 14. The Argo CD Dashboard After Deploying Applications

Figure 15 shows the detailed information: the application’s manifest and architecture, including ConfigMap, Secrets, Pods, PVC, and others.

Figure 15. The Detailed Information Page of an Application Deployed by Argo CD

Figure 16 shows the corresponding CNS volumes in vCenter, which aligns with the information in ‘reis-dev’. This page is in vCenter->Inventory->Datacenter->Cluster->Monitor->Cloud Native Storage->Container Volumes.

Figure 16. The ‘Container Volumes’ Monitoring Page in vCenter Showing the Corresponding PVC Created by redis-dev in Argo CD

Hardware Resources

In this solution, for the workload domain of Argo CD, we used a total of four VxRail R570F nodes. Each server was configured with two disk groups, and each disk group consisted of one cache-tier write-intensive SAS SSD and four capacity-tier read-intensive SAS SSDs.

Each VxRail node in the cluster had the following configuration, as shown in table 3.

Table 3. Hardware Configuration for VxRail

PROPERTY	SPECIFICATION
Server model name	VxRail P570F
CPU	2 x Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, 28 core each
RAM	512GB
Network adapter	2 x Broadcom BCM57414 NetXtreme-E 25Gb RDMA Ethernet Controller
Storage adapter	1 x Dell HBA330 Adapter
Disks	Cache - 2 x 800GB Write Intensive SAS SSDs Capacity - 8 x 3.84TB Read Intensive SAS SSDs

Software Resources

Table 4 shows the software resources used in this solution.

Table 4. Software Resources

Software	Version	Purpose
VMware Cloud Foundation on VxRail	4.3	A unified SDDC platform on VxRail that brings together VMware vSphere, vSAN, NSX, and optionally, vRealize Suite components, into a natively integrated stack to deliver enterprise-ready cloud infrastructure for the private and public cloud. See BOM of VMware Cloud Foundation on VxRail for details.
Dell VxRail	7.0.131	Turnkey Hyperconverged Infrastructure for hybrid cloud
VMware vSphere	7.0.2	VMware vSphere is a suite of products: vCenter Server and ESXi.
VMware vSAN	7.0.2	vSAN is the storage component in VMware Cloud Foundation to provide low-cost and high-performance next-generation HCI solutions.
NSX-T	3.1	NSX-T is the key network component in VMware Cloud Foundation on VxRail and is deployed automatically. It is designed for networking management and operation.
Argo CD	V2.2.0	The version of Argo CD software being tested in this solution.

Network Configuration

Figure 17 shows the VMware vSphere Distributed Switch^TM network configuration for Argo CD in the workload domain of the VMware Cloud Foundation on VxRail. NSX-T, which underlies the vSphere infrastructure, is used for the Argo CD and applications networking. To enable external access for the Tanzu Kubernetes cluster and its applications, an NSX-T edge cluster must be deployed. Also, it is required to configure the BGP peering and route distribution of the upstream network. For more details, refer to VMware Cloud Foundation 4.2 on VxRail Planning and Preparation Guide.

Figure 17. The Overall NSX-T Networking Architecture

Figure 17 shows the VMware vSphere Distributed Switches configuration for both the management domain and the workload domain of the VMware Cloud Foundation. For each domain, two 25 GbE vmnics were used and configured with teaming policies. The management domain can be shared among different workloads.

As shown in Figure 18, when Tanzu Kubernetes Clusters were created, some segments in NSX-T were automatically created.

Seg-domain-xxx is used for services in the supervisor cluster such as the embedded harbor private registry.
Vm-domain-xxx is used for supervisor cluster control plane virtual machines.
Vnet-domain-xxx is used for the Tanzu Kubernetes Cluster networking. Argo CD and its deployed applications use these segments. With more Tanzu Kubernetes Cluster being created, more segments called vnet-domain-xxx would be created.

Figure 18. Segments Created in NSX-T for Tanzu Kubernetes Clusters

The NSX-T controllers reside in the management domain. VMware vSphere vMotion®, vSAN, and VXLAN VTEP for NSX-T had another dedicated segment created.

Jumbo Frame (MTU=9000) was enabled on the physical switches, vSAN VMkernel, and all the virtual switches to improve performance.

NSX-T managers and edges have more than one instance to form NSX clusters to achieve HA and better load balancing. Besides, based on workloads, the vCPU and memory may be adjusted to achieve better performance. Table 5 shows the configuration of the NSX-T managers and edge nodes virtual machines. The NSX-T managers reside in the management workload domain, so it will not pull from the compute resources for the Argo CD workload domain. However, the NSX-T edge nodes reside in the Argo CD workload domain and it will cost some CPU and memory resources. This should be taken into consideration while doing the sizing of the cluster before Argo CD and any applications are deployed.

Table 5. NSX-T VM Configuration

nsx-T VM Role	INSTANCE	vCPU	memory (GB)	vm name	Virtual disk size	Operating System
NSX-T Manager	3	12	48	NSX-unified-appliance-<version>	200GB	Ubuntu
NSX-T Edge Nodes	2	4	8	Edge-<UUID>	120GB	Ubuntu

Best Practices

Use the High Availability version of Argo CD for deployment.
Use the same server model for the physical hosts in one workload domain.
Enable Jumbo Frame on the physical switches. Use Jumbo Frames on the vSAN VMKernel and all virtual switches.
Set Failures to Tolerate (FTT) to at least 1 in vSAN’s storage policy and the corresponding ‘storage class’ in Kubernetes for data protection.

Conclusion

VMware Cloud Foundation with Tanzu on VxRail delivers flexible, consistent, secure infrastructure and operations across private and public clouds. It is ideally suited to meet the demands of modern applications such as Argo CD and to support continuous delivery of modern applications.

With VMware Cloud Foundation with Tanzu, we can easily manage the lifecycle of the hybrid cloud environment. Besides, we have a unified management plane for all cloud native applications. With VMware Cloud Foundation, we can leverage the leading virtualization technologies including vSphere, NSX-T, and vSAN.

This solution paper demonstrated the architecture of running Argo CD with VMware Cloud Foundation with Tanzu on VxRail. We showed the configuration details, the hardware resources, and the software resources used in the solution validation. We showed the various configuration options in addition to the best practices. VxRail Manager and VMware Cloud Foundation Manager provided the lifecycle management. vSAN provides reliable, high-performance, and flexible storage to Argo CD. NSX-T provided the fine-grained, secured, and high-performance virtual networking infrastructure to Argo CD. Also, vSphere DRS and vSphere HA provided efficient resource usage and high availability. All the above lead to a consolidated solution of running Argo CD and providing continuous delivery functionality with VMware Cloud Foundation with Tanzu on VxRail.

References

VMware Cloud Foundation

o Announcing VMware Cloud Foundation 4.2

o Get the Facts of VMware Cloud Foundation – Part 6

About the Author

Victor (Shi) Chen, Solutions Architect in the Application Solutions team of the Cloud Infrastructure Business Group in VMware, wrote the original content. The following members also contributed to the doc review:

Ka Kit Wong, Staff Solutions Architect in the Application Solutions team of the Cloud Infrastructure Business Group in VMware
Myles Gray, Staff Technical Marketing Architect in VMware
Catherine Xu, Manager in the Application Solutions team of the Cloud Infrastructure Business Group in VMware
Vic Dery, Senior Principal Engineer of VxRail Technical Marketing in Dell