Argo CD as the Continuous Delivery Solution on VMware Cloud Foundation with Multi-Cloud Deployment

Executive Summary

Business Case

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. Argo CD can automatically control the cloud native applications’ deployment and manage their lifecycle. In addition, Argo CD is auditable, easy to understand, and easy to use.

Kubernetes is an orchestration platform that helps orchestrate containerized applications to run on a cluster of hosts. It is a system that automates the deployment and management of containerized applications on a given cloud platform or on-premises infrastructure.

VMware Cloud Foundation^TM is an effective way to run Kubernetes workloads at scale. VMware Cloud Foundation is the hybrid cloud platform for managing virtual machines and orchestrating containers, built on full stack hyperconverged infrastructure (HCI) technology.

With a single architecture that is easy to deploy, VMware Cloud Foundation can provision computing, networking, and storage on demand. VMware Cloud Foundation protects network and data with micro-segmentation and satisfies compliance requirements with data-at-rest encryption. Policy-based management delivers business-critical performance. VMware Cloud Foundation delivers flexible, consistent, secure infrastructure and operations across private and public clouds and is ideally suited to meet the demands of Kubernetes.

VMware Cloud on AWS is a jointly engineered service that brings VMware’s enterprise class Software-Defined Data Center SDDC software to the AWS Cloud, with optimized access to AWS services as well as robust disaster protection. With the same architecture and operational experience on-premises and in the cloud, IT teams can now quickly derive instant business value from use of the AWS and VMware hybrid cloud experience.

By using Argo CD in a VMware Cloud Foundation-based private cloud data center for development purposes and deploying production workloads into a public cloud, users can take advantage of the benefits of both environments. This approach can help you achieve better scalability, reliability, and cost-effectiveness for your applications.

In addition, by using some Disaster Recovery (DR) solutions such as Velero with Argo CD, users can back up the most important data and workloads to VMware Cloud on AWS and restore them from VMware Cloud on AWS. This approach can help users achieve better data protection and disaster recovery for their applications.

This solution provides the generic design and deployment guidelines for running and managing Argo CD on VMware Cloud Foundation and with Multi-Cloud support.

Business Values

Here are the top 5 benefits to deploy and manage Argo CD on VMware Cloud Foundation with Multi-Cloud (VMware Cloud on AWS) support:

• Rapid deployment and configuration: Native VMware Cloud Foundation deployment process of HCI infrastructure and SDDC Manager Orchestration in a single deployment workflow of VMware vSphere vCenter™, VMware NSX^®, and vRealize Suite above the VMware ESXi™ and VMware vSAN™ layers. An SDDC in VMware Cloud on AWS can also be provisioned fast and reliably both from UI and using API.
• Automated Lifecycle Management: Minimize the Argo CD and Kubernetes workload impact and downtime during necessary patching and upgrading of the full private and public cloud stack using automated and self-managed services within the workload domain.
• Providing developer-ready infrastructure, aligning DevOps and IT teams, and simplifying cloud operations. VMware Cloud Foundation also provides unified workload management for both virtual machine based and Kubernetes based.
• Unified management for computing, storage, and networking on the proven vSphere platform, both in the private cloud and in the public cloud.
• Continuous delivery of applications for both development and production environments, either deployed in the private cloud or in the public cloud. The continuous delivery is automated and seamless

Key Results

This reference architecture is a showcase of running and managing Argo CD on VMware Cloud Foundation and with Multi-Cloud support by connecting to VMware Cloud on AWS. Key results can be summarized as follows:

• VMware Cloud Foundation simplifies and accelerates the necessary virtual infrastructure deployment desired for Argo CD and Kubernetes workloads with a single workflow containing all individual sub-tasks.
• We demonstrated the overall architecture and configuration details used in this solution.
• We showed the detailed usage of continuous delivery by Argo CD both to the private cloud and public cloud.
• We demonstrated best practices used in this reference architecture.
• We showed the backup and restore options between the private and public clouds.

Note: The performance results in this solution are validated on the HCI platform of VMware Cloud Foundation, which is also applied to general VMware vSAN with similar configurations

Audience

This solution is intended for IT administrators, DevOps operators, and developers who are involved in the early phases of planning, design, and deployment of Argo CD on VMware Cloud Foundation and VMware Cloud on AWS. It is assumed that readers are familiar with the concepts and operations of Argo CD and VMware Cloud Foundation components such as vSphere, NSX, and vSAN.

Technology Overview

Solution technology components are listed below:

• VMware Cloud Foundation
• VMware Cloud on AWS
• Argo CD
• Monitoring Tools: Prometheus and Grafana
• Backup and Restore Tool: Velero

VMware Cloud Foundation

VMware Cloud Foundation is the ubiquitous hybrid cloud platform built on full stack HCI. Cloud Foundation provides a complete set of secure software-defined services for computing, storage, network, security, Kubernetes management, and cloud management. The result is an agile, reliable, and efficient cloud infrastructure that offers consistent infrastructure and operations across private and public clouds. See VMware Cloud Foundation 4.5 for detailed information.

VMware Cloud on AWS

VMware Cloud on AWS brings VMware’s enterprise class Software-Defined Data Center software to the AWS Cloud and enables customers to run production applications across VMware vSphere-based private, public and hybrid cloud environments, with optimized access to AWS services. Jointly engineered by VMware and AWS, this on-demand service enables IT teams to seamlessly extend, migrate and manage their cloud-based resources with familiar VMware tools –minimizes the hassles of learning new skills or utilizing new tools.

VMware Cloud on AWS integrates VMware’s flagship compute, storage, and network virtualization products (VMware vSphere, VMware vSAN and VMware NSX) along with VMware vCenter management as well as robust disaster protection, and optimizes it to run on dedicated, elastic, Amazon EC2 bare-metal infrastructure that is fully integrated as part of the AWS Cloud. This service is delivered and supported by VMware and its partner community. This service is sold by VMware, AWS, and their respective partners.

Argo CD

Argo CD is a declarative GitOps continuous delivery tool for Kubernetes. It follows the GitOps pattern of using Git repositories as the source of truth for defining the desired application state. Kubernetes manifests can be specified in several ways such as kustomize applications, helm charts, jsonnet files, plain directory of YAML/JSON manifests, or any custom config management tool.

Monitoring Tools

Prometheus is an open-source monitoring and alerting toolkit that is commonly used for Kubernetes but also supports other cloud-native environments. It is a high-scalable open-source monitoring framework that provides out-of-the-box monitoring capabilities for the Kubernetes container orchestration platform.

Grafana is an open-source analytics and monitoring platform that is commonly used for Kubernetes but also supports other cloud-native environments. It provides a powerful and elegant way to create, explore, and share dashboards and data with your team and others. Grafana can be used with Prometheus as a data source to visualize the metrics collected by Prometheus. Grafana can also be used with other data sources such as Graphite, Elasticsearch, InfluxDB, and more.

vSAN Performance Service is used to monitor the performance of the vSAN environment, using the vSphere web client. The performance service collects and analyzes performance statistics and displays the data in a graphical format. You can use the performance charts to manage your workload and determine the root cause of problems.

vSAN Health Check delivers a simplified troubleshooting and monitoring experience of all things related to vSAN. Through the vSphere web client, it offers multiple health checks specifically for vSAN including cluster, hardware compatibility, data, limits, and physical disks. It is used to check the vSAN health before the mixed-workload environment deployment.

Backup and Restore Tool: Velero

Velero is an open-source tool for safely backing up and restoring resources in a Kubernetes cluster, performing disaster recovery, and migrating resources and persistent volumes to another Kubernetes cluster. Velero offers key data protection features such as scheduled backups, retention schedules, and pre-backup or post-backup hooks for custom actions.

Solution Configuration

This section introduces the resources and configurations:

• Architecture diagram
• Hardware resources
• Software resources
• Virtual machine configuration

Architecture Diagram

In this solution, VMware Cloud Foundation in the local data center was composed of a management domain and a workload domain.

Typically, a VMware Cloud Foundation management domain can simultaneously manage multiple workload domains. All the workload domains’ management VMs, such as vCenter and VMware NSX Manager™, are placed in the management domain. All the workload domains can share one management domain.

Different workload domains can serve different business purposes. The one-to-many mapping simplifies the overall management of the whole VMware Cloud Foundation environment.

We deployed the Argo CD server and its deployed applications in one of the workload domains. The workload domain also contains the VMware NSX Edge nodes. All other infrastructure VMs were in the shared management workload domain (Figure 1).

Figure 1 shows the workload domain containing Argo CD that we focus on in this reference architecture. The management domain can manage multiple workload domains, but other workload domains are not shown in this figure.

Figure 1. The Overall Architectural Diagram of Argo CD on VMware Cloud Foundation Standard Architecture and with Multi-Cloud Support

The above architecture is called VMware Cloud Foundation’s standard architecture.

Note: Apart from the architecture in Figure 1, for a small test environment, the management domain and the workload domain can be consolidated into one domain. This architecture only requires a minimum of 4 ESXi hosts, which is ideal for a small or test environment. See https://docs.vmware.com/en/VMware-Cloud-Foundation/4.4/vcf-getting-started/GUID-C6AF75AE-569C-49F8-A15E-E9A6EF9549DA.html for details about VMware Cloud Foundation Consolidated Architecture Model.

Figure 2. The Overall Architectural Diagram of Argo CD on VMware Cloud Foundation Consolidated Architecture and with Multi-Cloud Support

In our solution, we tested both architectures.

Standard Architecture

For the standard architecture, we created a 4-node ESXi cluster for the VMware Cloud Foundation management domain, running management virtual machines and appliances. The management domain can be shared with other workload domains.

For the workload domain, we created another 4-node ESXi cluster with a separate NSX Fabric, deployed an NSX Edge cluster, and deployed the Argo CD server and its applications in the workload domain.

Consolidated Architecture

For consolidated architecture, the virtual machines and their roles are the same. The difference is that in consolidated architecture, there is only one cluster consisting of four ESXi hosts, and all virtual machines are in this cluster.

The solution architectures as shown in Figure 1 and Figure 2, either in Standard Architecture Model or Consolidated Architecture Model, is called a building block for a basic installation of Argo CD with VMware Cloud Foundation. Based on the customer demands and deployment size, we can expand the workload domain to include more physical hosts. A cluster with vSAN enabled supports up to 64 physical hosts for a non-stretched cluster. With adding more hosts to the vSAN cluster, not only the capacity of CPU and memory is increased for computing but also the capacity of vSAN storage is increased accordingly. This is one of the benefits of HCI that we can increase the capacity of computing and storage concurrently and proportionally.

Hardware Resources

In the private cloud VMware Cloud Foundation environment, each server has the following configuration as shown in Table 1. Four servers form the vSphere cluster in the private cloud as the workload domain.

Table 1. Hardware Configuration in Private Cloud

We used three Amazon EC2 ‘i3en.metal’ instances when deploying VMware Cloud on AWS SDDC. Each server has the following configuration.

Table 2. Hardware Configuration in the VMware Cloud on AWS

Refer to the feature brief about a more detailed description of i3en instances.

Software Resources

Table 3 shows the software resources used in this solution.

Table 3. Software Resources

Virtual Machine Configuration

See Table 4 for VMware Cloud Foundation virtual machine configuration used for management and infrastructure in this solution.

Table 4. Management Domain Virtual Machine Configuration

Solution Validation

Overview

This section validates how to run Argo CD in the VMware Cloud Foundation based private data center and connect with the VMware Cloud on AWS public data center, including Velero backup and restore scenario and component update considerations.

Installation and Networking Setting

vSphere with Tanzu Installation

Before installing Argo CD, firstly we must enable vSphere with Tanzu on both local VMware Cloud Foundation cluster and public VMware Cloud on AWS cluster.

Alternatively, we could also enable Tanzu Kubernetes Grid on both clusters. The installation is out of the scope of this paper. Check out the references for detailed information:

• vSphere with Tanzu Configuration and Management (vmware.com)
• VMware Tanzu Kubernetes Grid 1.6 Documentation

Argo CD Installation

We introduced the following three ways to install Argo CD. The methods and details are described below.

• Argo CD Operator

Argo CD Operator is a Kubernetes operator for managing Argo CD clusters. The operator manages the full lifecycle of Argo CD and its components. Its goal is to automate the tasks required when operating an Argo CD cluster. Beyond installation, the operator helps to automate the process of upgrading, backing up, and restoring as needed and remove the human as much as possible. It also aims to provide deep insights into the Argo CD environment by configuring Prometheus and Grafana to aggregate, visualize, and expose the metrics already exported by Argo CD.

Argo CD Operator provides the following benefits:

• Easy configuration and installation of the Argo CD components with sane defaults to get up and running quickly.
• Provide seamless upgrades to the Argo CD components.
• Ability to back up and restore an Argo CD cluster from a point in time or on a recurring schedule.
• Aggregate and expose the metrics for Argo CD and the operator itself using Prometheus and Grafana.

• Automate the tasks required when operating an Argo CD cluster. Beyond installation, the operator helps to automate the process of upgrading, backing up, and restoring as needed and eliminate the manual work as much as possible.

However, the Argo CD Operator is only v0.6.0 as of now. It is under active development, so use it with caution in a production environment.

Check out the Argo CD Operator installation reference: Getting Started - Argo CD Operator (argocd-operator.readthedocs.io)

• Argo CD Non-HA installation with a YAML file

The simplest way to install Argo CD for development purpose is to follow the Argo CD’s Getting Started document: Getting Started - Argo CD - Declarative GitOps CD for Kubernetes (argo-cd.readthedocs.io)

In this way, Argo CD is installed in a non-HA way, so the deployed pods have no replications. We only recommend using this method for testing or developing purpose but not using it for a production environment.

• Argo CD HA installation with a YAML file

We recommend using the HA installation method. In the YAML file, Argo CD has defined all the necessary roles, resources, and components.

The Non-HA installation method and the HA installation method will install the same containers, pods, and stateful sets. The only difference is that the HA installation method will create more replication for each component. In this way, it can achieve the goal of high availability.

The installation procedure is the same as the above Getting Started document: Getting Started - Argo CD Operator (argocd-operator.readthedocs.io)

Replace the used YAML file in the doc with:https://github.com/argoproj/argo-cd/blob/master/manifests/ha/install.yaml

Some key steps are as follows:

1. Create a role binding for the service account.

After we created the ‘argocd’ namespace in the Tanzu Kubernetes cluster, we must create a role binding for the service account with the ‘psp:vmware-system-privileged’ privilege. Otherwise, the deployment to the ‘argocd’ namespace would fail.

For example, create a file with the name ‘system-serviceaccounts-role-binding.yaml’,  which has the following content:

$ kubectl -n argocd apply -f system-serviceaccounts-role-binding.yaml

$ kubectl -n argocd apply -f argocd-ha-install-2-6-7.yaml

Using Argo CD to Connect to Kubernetes Cluster in the Local and in the Cloud

After Argo CD was successfully deployed, the ‘in-cluster’ Kubernetes cluster, in which the Argo CD server itself resides, was automatically added to Argo CD. Applications can be deployed to this cluster immediately. This was the local Kubernetes cluster in the local data center.

Since no applications were deployed to this cluster yet, the ‘connection status’ is shown as ‘unknown’.

Figure 8. The Initial View of the Argo CD Server’s Settings before Deploying Applications

Then, more external clusters can be added to Argo CD for deploying applications.

The external clusters can be any Kubernetes cluster whether they are other Tanzu Kubernetes clusters in the local data center or some Tanzu Kubernetes clusters in VMware Cloud on AWS.

As a showcase, we will add a Kubernetes cluster in VMware Cloud on AWS for demonstration. You can find more information about how to activate Tanzu Kubernetes Grid service on VMware Cloud on AWS using the URL: https://vmc.techzone.vmware.com/tanzu-kubernetes-grid-service-vmware-cloud-aws

Firstly, in a local Linux client machine, connect to the Tanzu Kubernetes Cluster in the VMware Cloud on AWS. For example:

$ kubectl vsphere login --server=k8s.Cluster-1.vcenter.sddc-x-x-x-x.vmwarevmc.com --vsphere-username cloudadmin@vmc.local --tanzu-kubernetes-cluster-name tkgs-argocd-cloud1-ns-prod-cluster --tanzu-kubernetes-cluster-namespace argocd-cloud1-ns

After successfully logging in, this command will give us the necessary kubectl context to operate the Kubernetes cluster in VMware Cloud on AWS. In our example, the cluster’s name is ‘tkgs-argocd-cloud1-ns-prod-cluster’ in the ‘argocd-cloud1-ns’ namespace.

Use the Kubectl command to retrieve the corresponding context:

$ kubectl config get-contexts -o name |grep -i prod

tkgs-argocd-cloud1-ns-prod-cluster

Then, use the following command to add the ‘tkgs-argocd-cloud1-ns-prod-cluster’ cluster to Argo CD.

$ argocd cluster add tkgs-argocd-cloud1-ns-prod-cluster

After adding the Kubernetes cluster from VMware Cloud on AWS, we can check the Argo CD UI to make sure that the cluster is added to the Argo CD server.

Since no applications are deployed to this cluster yet, the ‘connection status’ is shown as ‘unknown’ for both clusters.

Figure 9. The View of the Argo CD Server’s Settings after Adding an External Cluster in VMware Cloud on AWS

Application Deployment by Argo CD

We can use different Kubernetes clusters in Argo CD for different purposes. In this example, we used the local Kubernetes cluster as the target for development. Meanwhile, the Kubernetes cluster in VMware Cloud on AWS is used for production.

Firstly, we chose the Argo CD’s official example guestbook application for demonstration purposes: https://github.com/argoproj/argocd-example-apps.git

We used the name ‘guestbook-development’ for the development purpose. In the ‘+NEW APP’ page in Argo CD UI, input the application’s name, project name, and sync policy.

Figure 10. Deploy an Application using the Argo CD UI for Development in the Local VMware Cloud Foundation Data Center-Page1

Then input the application’s repository URL. We chose the revision ‘HEAD’ to indicate that the ‘main’ branch in a git repository is under active development. So, we deployed this branch to mimic a development environment. When people commit new codes into this branch, the application can be easily updated in the local Kubernetes cluster for development and debugging purpose.

For the application’s deployment destination, we chose ‘https://kubernetes,default.svc’ to indicate that it is a local Kubernetes cluster in the local data center. Users can also choose other local Kubernetes clusters in the local data center for development purposes.

Figure 11. Deploy an Application using the Argo CD UI for Development in the Local VMware Cloud Foundation Data Center-Page2

Then, we deployed the same application but with a stable version to VMware Cloud on AWS. This is used to publish a stable version of the application to serve a public service.

1. Firstly, we name it ‘guestbook-production’.

Figure 12. Deploy an Application Using the Argo CD UI for Production in VMware Cloud on AWS Data Center-Page1

2. On the next page, we chose a tag with ‘bg-guestbook-v0.1’ as an example. This code is not the HEAD in the main branch and is considered as a stable version. So, we can publish it to the public in VMware Cloud on AWS.

Use your own branches or tags in the code repository for your production deployment.

In the ‘destination’ page, we chose the Tanzu Kubernetes Cluster that resided in VMware Cloud on AWS.

Figure 13. Deploy an Application Using the Argo CD UI for Production in the Public VMware Cloud on AWS Data Center-Page2

3. After the application’s deployment was successfully done, we can check the Argo CD server’s UI to validate that the application has been deployed to two places: one for development in the local Data Center and another for production in VMware Cloud on AWS.

Figure 14. The Argo CD UI Shows the Applications Deployed to both Local VMware Cloud Foundation and Public VMware Cloud on AWS are Successful

4. After applications were successfully deployed, check the Clusters page in the Settings tab of the Argo CD UI. All the status of the added clusters turned from ‘unknown’ to ‘Successful’.

Figure 15. The Clusters’ Status Changed to ‘Successful’ after Applications were Deployed

Monitoring

We used Prometheus and Grafana for the monitoring of the Tanzu Kubernetes Clusters and monitoring of the pods and services on the clusters. https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.6/vmware-tanzu-kubernetes-grid-16/GUID-packages-user-managed-index.html

We followed the procedures for the installation of Prometheus and Grafana. There are dependencies for the installation. So, we need to install the following components in the following sequence:

1. Install cert-manager.
2. Implement Ingress Control with Contour.
3. Deploy Prometheus on Workload clusters.
4. Deploy Grafana on Workload clusters.

After Prometheus and Grafana were successfully deployed, we can use the query to monitor the workloads’ health.

For example, we can monitor the memory usage in the namespace ‘argocd’ by running the following query:

container_memory_working_set_bytes{namespace="argocd"}

Figure 16. Using Prometheus to Monitor the Applications’ Memory Usage in the namespace ‘argocd’

Grafana is connected to Prometheus and will take data from Prometheus. Grafana takes those data and graphically shows them. For example, in Grafana, we can create a new dashboard with the same query above. It shows the memory usage of the pods in the namespace ‘argocd’ in a graphic way.

Figure 17. Using Grafana to Visualize the Applications’ Memory Usage in the Namespace ‘argocd’

Similarly, we can create other queries to monitor other resources in the namespace ‘argocd’ or the node health in the Tanzu Kubernetes API.

If anything is abnormal during the monitoring, search for the respective documentation for the cascading troubleshooting.

• Troubleshooting Tanzu Kubernetes Grid Issues: Troubleshooting Tanzu Kubernetes Grid Issues (vmware.com)
• Troubleshooting Argo CD: Troubleshooting Tools - Argo CD - Declarative GitOps CD for Kubernetes (argo-cd.readthedocs.io)

Velero Backup and Restore

We used Velero to back up important Argo CD data and applications from the local data center to the public VMware Cloud on AWS. In case of a local failure, we can restore data from VMware Cloud on AWS to the local data center.

We followed this official document to install a standalone Velero server: https://docs.vmware.com/en/VMware-vSphere/7.0/vmware-vsphere-with-tanzu/GUID-A24A6B91-0CDF-4D02-AD08-7BA5EAC25A42.html

The steps are described below:

1. Create a new Linux-based virtual machine in VMware Cloud on AWS; for example, name it ‘MinIO-Server’.
2. Install the MinIO software in this ‘MinIO-Server’ following the above official document.
3. Install the Velero CLI in a local client machine.
4. Install Velero and Restic on the Tanzu Kubernetes cluster in the local VMware Cloud Foundation Data Center.

Run the following command for the upgrade:

$ velero install --image harbor-repo.vmware.com/dockerhub-proxy-cache/velero/velero:v1.9.6 --provider aws --plugins harbor-repo.vmware.com/dockerhub-proxy-cache/velero/velero-plugin-for-aws:v1.6.1 --bucket argocd-cluster-backup --secret-file /credentials-minio --use-restic --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://xxx.xxx.xxx.xxx:9000

Replace the docker image address with your own local registry address, such as harbor-repo.vmware.com in our example.

The version number v1.9.6 indicates a desired version of Velero. For successive operations, the version can be changed to a higher value for the upgrade. The ‘s3Url’ is the MinIO server’s address and port.

Then, Velero can be used to back up and restore data. Run the following command to back up all the applications, data, and persistent volumes in the ‘argocd’ namespace. Remember to replace the namespace for backing up some other namespaces.

$ velero backup create argocd-backup --default-volumes-to-restic --include-namespaces argocd --wait --log_dir ./velero-logs/ --log_file velero-log

The result looks like this:

Figure 18. Using Velero CLI to Check that the Backup is Successful

After the Velero command returns OK, we can check the MinIO UI in the VMware Cloud on AWS Data Center to validate that all the data are successfully stored in the public cloud.

Figure 19. Checking the MinIO UI to Validate that the Backup Data is Successfully Stored in Public VMware Cloud on AWS

To test the restore of the data:

1. Firstly, delete the ‘argocd’ namespace.

$ kubectl delete ns argocd

The “argocd” namespace was deleted.

2. Then, use the Velero CLI to create a restore job named ‘argocd-restore-1’.

vmware@ubuntudt:~/velero$ velero restore create argocd-restore-1 --from-backup argocd-backup

The "argocd-restore-1" restore request was submitted successfully.

Run ‘velero restore describe argocd-restore-1’ or ‘velero restore logs argocd-restore-1’ for more details.

3. After a while when the restore completes, we can check the resources in the namespace ‘argocd’ are indeed successfully restored. For example, check the pods, stateful sets, and replica sets. Users can also check other resources based on their applications.

Figure 20. Validate that the Restore is Successfully Completed

Updates and Upgrades

There are different layers and software components that need to be updated or upgraded during the whole lifecycle management. We will demonstrate each component below.

• VMware Cloud Foundation

For VMware Cloud Foundation, we recommend using the SDDC manager’s lifecycle manager UI for the updates and upgrades. The UI is easy to use and straightforward. It handles the update and upgrade of all the VMware Cloud Foundation components, including SDDC manager, vCenter, ESXi server, vSAN, and NSX Data Center.

Locate the workload domain in the SDDC manager UI and find the Updates/Patches page if there are updated version available. Users can easily update all the software components here by going through the guided steps from UI starting from Precheck.

Figure 21. The SDDC Manager UI of VMware Cloud Foundation for Updates/Patches

• SDDC in VMware Cloud on AWS

When a new version of the SDDC in VMware Cloud on AWS is available, customers can raise a ticket to get a new version, but it is not guaranteed. The upgrade is no disruptive to the application workload. To minimize possible performance impact, an additional and non-billable host is added to the SDDC. Find more information here: https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws-operations/GUID-EE89B216-BE93-4A1A-9280-8F20E2A5266F.html.  Argo CD users typically do not directly involve in the process of upgrading the SDDC in the public VMware Cloud on AWS.

• Tanzu Kubernetes clusters

In the initial deployment, we used a YAML file to define the Tanzu Kubernetes cluster. If we want to upgrade the cluster, we can modify the version in the ‘tkr’ field as shown in Figure 22. Meanwhile, we can also modify this file, such as the number of the control plane nodes or the number of the worker nodes and other fields to update the cluster to meet new requirements.

Figure 22. Upgrading the Tanzu Kubernetes Clusters by Editing the Configuration File

• Argo CD server upgrade

For the Argo CD server upgrade, we just need to check the updated version of the YAML file we used in the deployment. For example, if we used an initial version of V2.6.6, follow the URL:

https://github.com/argoproj/argo-cd/blob/v2.6.6/manifests/ha/install.yaml

For the upgrade, replace it with the V2.6.7 version:

https://github.com/argoproj/argo-cd/blob/v2.6.7/manifests/ha/install.yaml

We can use some file comparison tools to compare the differences between versions. Pay attention to the image version and some configuration fields’ changes. Also, replace the image with your local image registry address if necessary.

Then, just run the same command for the upgrade as that is for installation:

$ kubectl -n argocd apply -f argocd-ha-install-2-6-7.yaml

• Velero server upgrade

For the Velero server upgrade, we need to perform the following procedures:

1. Upgrade the MinIO server in VMware Cloud on AWS.

2. Run the following command to get a newer version of MinIO if there is any:

           wget https://dl.min.io/server/minio/release/linux-amd64/minio

3. Restart the menu server.

4. Upgrade the Velero CLI.

5. Download the supported version of the signed Velero binary for vSphere with Tanzu from the VMware product downloads page.

6. Upgrade the Velero server.

7. Run the following command for the upgrade:

$ velero install -image harbor-repo.vmware.com/dockerhub-proxy-cache/velero/velero:v1.9.7 –provider aws –plugins harbor-repo.vmware.com/dockerhub-proxy-cache/velero/velero-plugin-for-aws:v1.6.1 –bucket argocd-cluster-backup –secret-file ./credentials-minio –use-restic -backup-location-config region=minio,s3ForcePathStyle=”true”,s3Url=http://xxx.xxx.xxx.xxx:9000

• Replace the docker image address with your own local registry address, such as harbor-repo.vmware.com in our example.
• The version number v1.9.7 indicates a newer version of Velero. Replace it with the correct version.
• The ‘s3Url’ is the MinIO server’s address and port.

Best Practices

In this solution, we validated the Argo CD as the continuous delivery solution on VMware Cloud Foundation as well as connecting and deploying applications to VMware Cloud on AWS.

The following recommendations provide the best practices to run Argo CD on VMware Cloud Foundation and VMware Cloud on AWS.

• Cloud management consideration:
  • Use VMware Cloud Foundation SDDC Manager for the initial deployment of the local VMware Cloud Foundation data center for an automatic bring-up of the infrastructure.
  • Use VMware Cloud Foundation SDDC Manager for the entire lifecycle management, including vCenter, ESXi servers, vSAN, and NSX Data Center. This eases the update/upgrade of the infrastructure.
• Network considerations:
  • Separate physical network using different network ranges and VLAN IDs for management, VMware vSphere vMotion®, vSAN, VM network, and VXLAN VTEP network.
  • For NSX Data Center for vSphere design best practices for workload domain, see VMware NSX for vSphere (NSX) Network Virtualization Design Guide.
  • Set up proper networking connections between the local data center and the public VMware Cloud on AWS data center; for example, set up VPN, AWS Direct Connect, or NATs for the bi-directional traffic between them.
• Storage consideration:
  • For different types of workloads, create a different vSAN storage policy for management and workloads. Then, bind them to different storage classes. Specifically, on VMware Cloud on AWS make sure to use a default management policy to stay compliant with VMware Cloud on AWS SLAs. if using custom policies, follow the SLA guidelines when defining the RAID level and FTT for the workloads.
• Argo CD consideration:
  • Create more Tanzu Kubernetes clusters for different types of workloads or when a Kubernetes cluster has been saturated.
  • Enable the monitoring tools such as Prometheus or Grafana to monitor the health and events of Argo CD and its deployed applications. Remediate the errors if there are any during the monitoring.
  • Use a backup and restore solution such as Velero to back up the data to the VMware Cloud on AWS. Restore the data from the public VMware Cloud on AWS if there is any site failure or an unrecoverable error

Conclusion

In this solution, we demonstrated how to run Argo CD in the VMware Cloud Foundation based private data center and in VMware Cloud on AWS. Firstly, we demonstrated the overall architecture and the required hardware and software resources in this reference architecture. Then we explained the detailed steps of configuring each software component. Following that, we showed how to deploy applications by using Argo CD both to the local data center for development purpose and deploying applications to the public VMware Cloud on AWS for production. We also showed how to back up data to the public cloud and restore data from the public cloud. This reference architecture provides guidance to IT administrators, developers, and DevOps engineers on how to use Argo CD with Multi-Cloud support.

Reference

• VMware Cloud Foundation
• VMware vSphere
• VMware vSAN
• VMware NSX Data Center
• VMware Cloud on AWS
• Argo CD

About the Author

Victor (Shi) Chen, Sr. Technical Marketing Manager in the Workload Technical Marketing Team of the Cloud Infrastructure Big Group, VMware Inc, wrote the original version of this paper.

The following reviewers also contributed to the paper’s contents: 

• Oleg Ulyanov, Staff Cloud Solutions Architect and Group Manager at VMware

• Catherine Xu, Senior Manager of the Workload Technical Marketing team at VMware