vSAN Proof of Concept: vSAN Architecture Overview & Setup

Introduction

Overview

This document primarily details the overall architecture and setup of vSAN Express Storage Architecture™ (ESA) cluster environments. vSAN Original Storage Architecture™ (OSA) environments are covered where they differ from vSAN ESA.   The specific focus of the document includes overviews of:

• Single vSAN HCI cluster deployments (ESA and OSA)
• vSAN Max™ cluster deployments (disaggregated storage)
• vSAN Compute cluster deployments
• vSAN Two-Node and Stretched Cluster deployments.

Additionally, the document includes reviews of basic functions such as vSAN storage policies and virtual machine deployment.

Separate guides are dedicated to more detailed discussions of specific vSAN related areas:

• vSAN Proof of Concept: vSAN Management, Monitoring & Hardware Testing
• vSAN Proof of Concept: vSAN Features
  • Space efficiency features (e.g., compression, deduplication, RAID-5/RAID-6 erasure coding, and Trim/Unmap)
  • Encryption
  • File Services
• vSAN Proof of Concept: vSAN Performances Testing
• vSAN Proof of Concept: vSAN Stretched Cluster & Two-Node Overview & Testing

What’s New in vSAN 8.0

vSAN 8.0 is a major milestone release that includes a new architecture option for new vSAN clusters, and there are now two distinct ways to configure a vSAN cluster. vSAN 8.0 introduced ESA, which optimizes the use of certified NVMe drives in a single tier, with a dynamic approach to data placement. vSAN 8.0 Update 2 introduces Max. A disaggregated deployment model, providing scalable, cost-effective storage solutions to empower your cloud infrastructure. This innovative approach caters to three primary use cases: driving cost optimization for infrastructure and applications, ensuring operational simplicity with unified storage, and facilitating rapid scaling for cloud-native applications. The classic OSA model, which is updated from vSAN 7, remains an option.

For a comprehensive overview of ESA, visit: https://core.vmware.com/vsan-esa.

• Updates to the OSA include:
• Change in buffer device capacity, increasing from 600GB to 1.6TB.
• For HCI Mesh, the number of server and client clusters increases from 5 to 10
• Improvements in vSAN File Services operations
• vSAN Boot time optimizations

Hardware Selection

Choosing the appropriate hardware is one of the most important factors in the successful validation of vSAN. Hardware recommendations will differ between ESA and OSA deployments.

There are many variables with hardware (drivers, controller firmware versions) so be sure to choose hardware that is on the VMware Compatibility List. See the ‘Prerequisites’ section of this guide for more information.  

General Approach to Testing vSAN

Once the appropriate hardware is selected for testing, it is useful to define the use case, goals, expected results and success metrics.

The testing lifecycle can be broken into three phases:

Day-0

The post-design phase, installation of the infrastructure components, including: the hypervisor VMware ESXi™ (ESXi); control plane VMware vCenter®(vCenter); physical network uplinks and upstream network devices; essential services, such as DNS and NTP; VLANs, etc. 

Day-1

Setup and configuration of the required solution (in the case of vSAN).

Day-2

Operational aspects and monitoring.  The most important aspects to validate when testing:

• Successful vSAN configuration and deployment
• VMs successfully deployed to vSAN Datastore
• Reliability: VMs and data remain available in the event of failure (host, disk, network, power)
• Serviceability: Maintenance of hosts, disk groups, disks, clusters
• Performance: vSAN and selected hardware can meet the application, as well as business needs
• Validation: vSAN features are working as expected (File Service, Deduplication and Compression, RAID-5/6, checksum, encryption)
• Day-2 Operations: Monitoring, management, troubleshooting, and upgrades

These can be grouped into three common types: resiliency testing, performance testing, and operational testing.

Resiliency Testing

As with any storage solution, failures can occur on hardware components at any time due to age, temperature, firmware, etc. Such failures can occur among storage controllers, disks, nodes, and network devices among other components. As a software solution, vSAN is designed to be resilient against these failures. In this guide, we will examine how to systematically test against disk, host, network, and control plane failures.

Operational Testing

Understanding how the solution behaves during normal (or “day two”) operations is important to consider as part of the evaluation. Fortunately, because vSAN is embedded within the ESXi hypervisor, many vSAN operations tasks are simply extensions of normal VMware vSphere® (vSphere) operations. Adding hosts, migrating VMs between nodes, and cluster creation are some of the many operations that are consistent between vSphere and vSAN.

Performance Testing

Before embarking on testing, it is important to set clear objectives and understand the performance requirements of the environment. Close attention to details such as workload I/O profiles, latency and hotspots is needed. In this guide, we will explore how to conduct performance tests with a consistent, methodological approach.

Prerequisites

Hardware Compatibility

Plan on testing a reasonable hardware configuration resembling a production-ready environment that suits your business requirements. Refer to the VMware vSAN Design and Sizing Guide for information on design configurations and considerations when deploying vSAN. 

As vSAN is a software solution, it is critical to ensure that well supported, enterprise class hardware components are used. The VMware Compatibility Guide (or “VCG”) lists components that have been tested and certified for use with vSAN. In addition, the vSAN Hardware Compatibility List (or “HCL”) lists hardware compatibility specific to vSAN.

Note: The terms “VCG” and “HCL” may be used interchangeably (both within this guide, and in other documentation), but essentially pertain to hardware compatibility. 

Note: If using a VMware vSAN ReadyNode™ (ReadyNode) or appliance, the factory-installed hardware is guaranteed to be compatible with vSAN. However, be mindful that BIOS updates and firmware and device driver versions may be out of date and should be checked.

vSAN ESA or OSA

If using a certified ReadyNode or vSAN appliances with specific certified devices, then vSAN ESA (which has an optimized data path and placement for NVMe devices) is an option.

The table below summarizes the minimum requirements for each architecture:

Note that 25Gbps networking (or faster) is required vSAN ESA. The one exception is the vSAN-ESA-AF-0 ReadyNode configuration, which allows for 10Gbps networking. It is designed for Edge, 2-node, and other small environments with relatively few VMs, and minimal workload demands.  For more information, please refer to: https://core.vmware.com/blog/esa-all-platforms-and-all-workloads-esa-af-0

vSAN ESA ReadyNode™ & ReadyNode Emulated Configuration Guidance

Customers can pick a server manufacturer’s ESA certified ReadyNode platform listed on the VMware Compatibility Guide (VCG) for vSAN ESA, and can build out a server configuration using hardware components as they are listed on the given ReadyNode specification, emulating a ReadyNode purchased using a single SKU.  This approach can help customers who chose not to purchase a ReadyNode through an official SKU but have the same or better hardware found in the desired ReadyNode classification. 

Here is the direct link to the VMware Compatibility Guide vSAN ESA page: https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsanesa

For further vSAN ESA hardware guidance, see:

• vSAN ESA ReadyNode Hardware Guidance -  https://www.vmware.com/resources/compatibility/vsanesa_profile.php
• KB article on vSAN ESA architecture - https://kb.vmware.com/s/article/90343
• Support for ReadyNode Emulated Configurations in vSAN ESA Blog - https://core.vmware.com/blog/support-readynode-emulated-configurations-vsan-esa

vSAN OSA Hardware Guidance

As with ESA deployments, OSA supports both prescriptive ReadyNode and customized architectures. Below is the direct link to the VMware Compatibility Guide vSAN OSA page:

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsanosa

For further vSAN OSA hardware guidance, see:

• vSAN OSA Hardware Quick Reference Guide -  https://www.vmware.com/resources/compatibility/vsan_profile.html?locale=
• KB article on vSAN OSA architecture - https://kb.vmware.com/s/article/52084

vSAN Setup Assumptions

This document assumes a generic vSphere deployment (one vCenter and at least four ESXi hosts).  This document does not assume the test environment is deployed using the VMware Cloud Foundation™ (VCF) model. If the VCF model is used, there may be an impact to the information in this guide. Those impacts are noted and references to the proper VCF versions are provided.

The list below details the setup requirements prior to using this guide:  

• Hosts with supported hardware and firmware (see above)
• All hosts with ESXi 8.0 Update 2 (build number 22380479) or newer deployed
• vCenter Server 8.0 Update 2 (build number 22617221) or newer has been deployed to manage these ESXi hosts
• DNS and NTP services are available 
• The hosts should not be added to any cluster
• If using the Quickstart workflow
  • Each host should have a management network VMkernel
  • VMware vSphere® vMotion (vMotion) and vSAN networks will be configured during the workflow
• If not using the Quickstart workflow, each host should have a management and a vMotion network VMkernel port already configured
• A set of IP addresses, 
  • One per ESXi host will be needed for the vSAN network VMkernel ports (the recommendation is that these are all on the same VLAN and IPv4 or IPv6 network segment)
  • One per ESXI will be needed for the vMotion network VMkernel ports (the recommendation is that these are all on the same VLAN and IPv4 or IPv6 network segment)
• If possible, configure internet connectivity for vCenter such that the HCL database can be updated automatically (Internet connectivity is also a requirement to enable Customer Experience Improvement Program or CEIP)
• Optionally, for the purposes of testing Storage vMotion operations, an additional datastore type (such as NFS or VMFS) should be presented to all hosts

vSAN OSA All-Flash or Hybrid

When reviewing whether to deploy all-flash or hybrid configurations (all-flash recommend), there are several factors to consider:

• All-Flash vSAN requires at least a 10Gb Ethernet network 
• Flash devices are used for both cache and capacity
• Deduplication and Compression are space-efficiency features available in all-flash configuration and not available with hybrid configuration 
• Erasure Coding (RAID 5/ RAID 6) is a space efficiency feature available on all-flash configuration only
• Flash read cache reservation is not used with all-flash configurations; reads come directly from the capacity tier SSDs
• Endurance and performance classes now become important considerations for both cache and capacity layers

Enabling a Single vSAN HCI Cluster

Using Quickstart to Enable Single vSAN HCI Cluster

Follow this section to configure a single vSAN HCI cluster via the Quickstart process (recommended). 

Note:

• If you wish to manually setup a vSAN HCI Cluster, please refer to the Manually Enabling vSAN Services on a Cluster section
• If you deployed vSAN using the bootstrap feature of vCenter deployment, you will not be able to use Quickstart to configure your environment
• vSAN Max deployments, disaggregated storage, requires a specific vSAN ESA deployment, if you plan to test vSAN Max, please skip this section a go directly to the Enabling vSAN Max - Disaggregated Storage Section  
• ESA and OSA HCI deployment steps are similar
• The walkthrough below calls out where OSA and ESA steps may differ

Initialize Cluster

Navigate to your Datacenter > Click Actions > New Cluster.

The New Cluster screen pops-up and we are presented with a dialog to enable services. Provide a name for the cluster and select vSAN from the list of services. We are also able to enable vSAN ESA (the default).

Note: Once the cluster is created with the ESA flag, it cannot be changed unless the cluster is re-created.

Be aware that:

• If targeting an OSA deployment, the “Enable vSAN ESA” checkbox must be unchecked
• For the Quickstart workflow to configure the vMotion VMkernel, vSphere DRS must be set to enabled

 vSAN ESA Example:

vSAN OSA Example:

We can also setup the cluster to use a single image (thereby enabling vLCM). For more information on vLCM, see: https://core.vmware.com/resource/introducing-vsphere-lifecycle-management-vlcm.

Quickstart - Cluster Basics

The initial cluster creation above initializes the Quickstart process. Once the cluster has been created, navigate to [vSAN Cluster] > Configure > Quickstart.  On this screen you will be able to confirm the basic services selected previously then move to the add hosts and configuration phases.

vSAN ESA Example:

vSAN OSA Example:

Quickstart - Add Hosts

The next step is to add hosts. The process for ESA and OSA is identical, clicking on the 'Add' button on the 'Add hosts' section presents the dialog below. Multiple hosts can be added at the same time (by IP or FDQN). Additionally, if the credentials of every host are the same, tick the checkbox above the list to quickly complete the form. If the hosts are already in vCenter, they will appear in the ‘Existing hosts’ tab.

Note: It may be useful to leave one host out of the configuration here to later demonstrate cluster expansion.

Once the host details have been entered, click Next. You are then presented with a dialog showing the thumbprints of the hosts. If these are as expected, tick the checkbox(es) and then click OK.

A summary will be displayed, showing the vSphere version on each host and other details. Verify the details are correct:

On the next screen, we have the option to import the ESXi an image from a host (to set as the cluster’s new image). Select an option and continue.

Finally, review and click Finish if everything is in order.

After the hosts have been added, validation will be performed automatically on the cluster. Check for any errors and inconsistencies and re-validate if necessary.

Quickstart - Configure Cluster

The next step is to configure the vSAN HCI cluster. After clicking on Configure under Step 3: Configure Cluster, the Configure Cluster workflow will start.  For ESA deployments the first step will ask whether this is a vSAN HCI or vSAN Max deployment. Ensure the vSAN HCI radio button is selected.  OSA deployments do not have this screen. Instead, they start at the Distributed Switches step.

Step 1: Select Cluster Type:

Step 2: Configure Distributed Switches:

The next dialog allows for the configuration of the distributed switch(es) for the cluster. Leave the default 'Number of distributed switches' set to 1 and assign a name to the switch and port groups.

Scroll down and configure the port groups and physical adapters as needed.

Steps 3 & 4: Configure vMotion and Storage Traffic:

On the next two screens, set the VLAN and IP addresses to be used for the vMotion and vSAN for each host. The ‘autofill’ function can be used to input consecutive IP addresses.

Step 5: Configure Advanced Options:

Here we select ‘Single site cluster’ as the deployment type. For stretched clusters, refer to the stretched cluster guide 

Auto-Policy Management is a vSAN ESA specific feature that ensures resilience settings for the environment are optimally configured automatically. Enabling this feature does not preclude the creation of custom storage policies as needed post deployment. For this deployment Auto-Policy management is set to the default “enabled.”

Note: OSA deployments do not support Auto-Policy Management or the RDMA support features. These options will be grayed out in an OSA deployment. 

For more information please review: https://core.vmware.com/blog/auto-policy-remediation-enhancements-esa-vsan-8-u2

Ensure that an NTP server is added to the configuration (under ‘Host Options’). As vSAN is a distributed system, features may not work as expected if there is a time drift between servers (ideally, also ensure that vCenter has the same time source).

Step 6: Claim Disks:

On the next screen, select the disks to use. For a vSAN OSA cluster, the system will attempt to select disks automatically as cache or capacity (on a vSAN ESA cluster, there is no tier selection). As the ESA example images indicate, the workflow checks drive compatibility and warns if an issue is detected.

vSAN ESA example:

 vSAN OSA Example:

Step 7: Review:

Finally, check everything is as expected and click Finish.

Monitor the creation in the task view and wait for the cluster to be formed.

Using Quickstart, we have created a cluster with vSAN enabled with the correct networking in place. 

Manually Enabling vSAN Services on a Cluster

Note: If Quickstart was used (as per the earlier section) then this section can be skipped.

Manual vSAN enablement is available for those that do not wish to use the Quickstart process.

For this scenario, please follow the Manually Enabling vSAN instructions on the VMware Docs page linked below: https://docs.vmware.com/en/VMware-vSphere/8.0/vsan-planning/GUID-53571374-B3E5-4767-A372-FEB7C995AF71.html

Enabling vSAN Services on a VMware Cloud Foundation™ based Cluster

VCF includes dedicated processes to automate the deployment and configuration of core infrastructure including vSAN services.   In fact, these processes are required and are the only supported methods within VCF.  This applies to standing up the initial Management Domain, a subsidiary Workload Domain, or a cluster with a Domain.

For more information on enabling vSAN services within the VCF Model please review these links:  

• vSAN Storage with VCF – https://docs.vmware.com/en/VMware-Cloud-Foundation/5.1/vcf-admin/GUID-766FBED1-6FE8-46D0-99C5-62355478F2CF.html
• VCF Admin Guide main page – https://docs.vmware.com/en/VMware-Cloud-Foundation/5.1/vcf-admin/GUID-D5A44DAA-866D-47C9-B1FB-BF9761F97E36.html
• Management Domain Deployment Walkthrough – https://core.vmware.com/cloud-foundation-automation#domain-operations
• Workload Domain Deployment Walkthrough -https://core.vmware.com/cloud-foundation-automation#domain-operations
• Add a Cluster to an existing Workload Domain – https://core.vmware.com/cloud-foundation-automation#cluster-operations

Enabling vSAN Max™ - Disaggregated Storage

What is vSAN Max? VMware’s new disaggregated storage offering that provides Petabyte-scale centralized shared storage for your vSphere clusters. 

Built on the foundation of vSAN ESA, vSAN Max is a fully distributed architecture, where access to data is NOT funneled through centralized I/O controllers but using the full power of each node (host) in the vSAN Max cluster.  The aggregated resources across all hosts in a vSAN Max cluster contribute to the processing of I/O.  The addition of new hosts means that it can scale capacity and performance linearly.

For more additional technical information please review:

• Main vSAN Max informational page – https://core.vmware.com/vsan-max
• vSAN Max Design & Operations Guide – https://core.vmware.com/resource/vsan-max-design-and-operational-guidance
• vSAN Max introduction blog post – https://core.vmware.com/blog/introducing-vsan-max
• vSAN Max scalability blog post – https://core.vmware.com/blog/vsan-max-and-advantage-scalability

vSAN Max Sizing Considerations

When sizing vSAN Max deployments, consider that vSAN Max clusters support:

• A maximum of 32 ESXi hosts in the cluster (24 ESXi hosts recommended)
• A maximum of 10 compute clusters mounting to a single vSAN Max cluster
• A maximum of 128 total ESXi hosts (both within the vSAN Max cluster and the vSAN Compute clusters connecting to a single vSAN Max datastore)

Note: Limiting the vSAN Max cluster size to 24 ESXi hosts will allow for up to 104 ESXI hosts from vSAN compute clusters to mount the datastore, offering a 4.3:1 ratio. A vSAN Max cluster size of 32 ESXi hosts would allow for up to 96 ESXI hosts from vSAN compute clusters to mount the datastore, offering a storage ratio of 3:1.

Disaggregated Storage for vSAN OSA (AKA: HCI Mesh)

Although the vSAN Max is explicitly a vSAN ESA function, vSAN OSA deployments still support disaggregated storage. 

vSAN OSA datastores can be shared between two vSAN clusters, utilizing vSAN’s native data path for cross-cluster connections. Compute Only Clusters are also supported.

Each vSAN OSA client cluster can mount a maximum of ten remote vSAN OSA datastores. A vSAN OSA server cluster can export its datastore up to a maximum of ten client clusters.

All vSAN features are supported except for Data-in-Transit encryption, Cloud Native Storage (including vSAN Direct), Stretched Clusters, and 2-Node Clusters. Additionally, HCI Mesh will not support remote provisioning of File Services Shares, iSCSI volumes, or First-Class Disks (FCDs). File Services, FCDs, and the iSCSI service can be provisioned locally on clusters participating in a mesh topology but may not be provisioned on a remote vSAN datastore.

The same MTU sizing is required for both the Client and Server clusters.

Using Quickstart to Enable vSAN Max Cluster

vSAN Max leverages vSAN ESA, as such the initial enablement process is very similar to the steps reviewed in the Using Quickstart to Enable Single vSAN HCI Cluster section of this document.

Initialize Cluster

Navigate to your Datacenter > Click Actions > New Cluster.

The New Cluster screen pops-up and we are presented with a dialog to enable services. Provide a name for the cluster and select vSAN from the list of services. Ensure that vSAN ESA (the default). For the Quickstart workflow to configure the vMotion VMkernel, vSphere DRS must be set to enabled.

We can also setup the cluster to use a single image (thereby enabling vLCM). For more information on vLCM, see: https://core.vmware.com/resource/introducing-vsphere-lifecycle-management-vlcm.

Quickstart – Cluster Basics

The initial cluster creation above initializes the Quickstart process. Once the cluster has been created, navigate to [vSAN Cluster] > Configure > Quickstart. On this screen you will be able to confirm the basic services selected previously then move to the add hosts and configuration phases. 

Quickstart – Add Hosts

The Adding Hosts steps for vSAN Max are identical to those for a single vSAN HCI cluster.  Refer to the  steps documented in the Enable a Single vSAN HCI, Quickstart – Add Hosts section of this document.

Quickstart – Configure Cluster

The next step is to configure the vSAN Max cluster. After clicking on Configure under Step 3: Configure Cluster, the Configure Cluster workflow will start.  Ensure that vSAN Max is selected.

Step 1: Select Cluster Type

Steps 2 -7: Configuring the Cluster

The remaining steps to configure the vSAN Max cluster are identical to those for a single vSAN HCI cluster. Refer to the steps documented in the Enable a Single vSAN HCI, Quickstart – Configure Cluster section of this document.

After the new vSAN Max cluster creation completes, navigate to [vSAN Cluster] > Configure > vSAN > Services. The screen will show that the vSAN Max cluster is ready to provide disaggregated storage to vSAN Computer clusters.

Now navigate to [vSAN Cluster] > Configure > vSAN > Remote Datastore.  This screen shows the name of the remote datastore created by the vSAN Max cluster configuration workflow.  The datastore name is a default name. If you wish to rename this datastore please refer to the Post-Configuration – Renaming vSAN Datastore section of this document.

If you do not wish to rename the datastore, you are now ready to configure a vSAN Compute cluster and mount this datastore to the Compute cluster. Go to the Enabling vSAN Compute Cluster section. 

 Post-Configuration – Renaming vSAN Datastore (optional)

Once the vSAN Max cluster creation completes, the vSAN Max datastore is ready to be shared with vSAN Compute Clusters.  The datastore will have the default name of “vsanDatastore.” If the default name is not suitable for your environment, use these steps to rename the datastore as needed.

Navigate to [vSAN Cluster] > Datastores. Once on that screen, filter on vSAN (to make it easier to find the new datastore otherwise one may see the local datastore for each cluster host as well).  Then right-click on the vSAN datastore and select Rename.

This will open the Rename workflow. In the workflow rename the datastore as needed then select OK

The datastore will now reflect the newly assigned name.

You are now ready to configure a vSAN Compute cluster and mount this datastore to the Compute cluster. Go to the Enabling vSAN Compute Cluster section. 

Manually Enabling vSAN Max on a Cluster

Note: If Quickstart was used (as per the earlier section) then this section can be skipped.

Manual vSAN Max enablement is available for those that do not wish to use the Quickstart process.

For this scenario, please follow the vSAN Max Cluster Services Configuration instructions in the vSAN Max Design and Operational Guidance document. Direct link to the section listed below:

https://core.vmware.com/resource/vsan-max-design-and-operational-guidance – sec32263-sub1

Enabling vSAN Max/HCI Mesh Services on a VMware Cloud Foundation™ based Cluster

VCF includes dedicated processes to automate the deployment and configuration of core infrastructure including vSAN services.   In fact, these processes are required and are the only supported methods within VCF.

As of the writing of this guide, VCF 5.1 supports HCI Mesh. For more information, please review below.

HCI Mesh with VCF - https://docs.vmware.com/en/VMware-Cloud-Foundation/5.1/vcf-admin/GUID-1F86850D-E95E-40A8-AFC5-BE58D504D739.html
 

Enabling vSAN Compute Cluster 

Overview

A vSAN Compute cluster is simply a vSphere cluster that has a thin layer of vSAN installed for the purposes of mounting the remote vSAN Max datastore. 

The configuration process is solely manual. We do not have a Quickstart process for configuring vSAN Compute clusters.  For this scenario, please follow the vSAN Compute Cluster instructions in the vSAN Max Design and Operational Guidance document.  Direct link to the section: https://core.vmware.com/resource/vsan-max-design-and-operational-guidance#sec32263-sub2

Prerequisites:

• The hosts in the vSAN Compute cluster will need a vSAN VMkernel configured (routable to the vSAN network used by the target vSAN Max cluster)
• If the vSAN Compute cluster is initialized as a vSAN ESA cluster, it will only mount vSAN ESA/vSAN Max datastores.
• If the vSAN Compute cluster is initialized as a vSAN OSA cluster, it will be able to mount vSAN OSA as well as vSAN ESA/Max datastores
• If direct access to the datastore is not required, one can configure vSAN File Services
• For more information on configuring vSAN File Shares refer to the “vSAN Proof of Concept: vSAN Features" guide

Enabling vSAN Compute Clusters in VMware Cloud Foundation

VCF includes dedicated processes to automate the deployment and configuration of core infrastructure including vSAN services.   In fact, these processes are required and are the only supported methods within VCF.  

As of the writing of this guide, VCF 5.1 supports vSAN Compute clusters. For more information, please review below:

• Commissioning Hosts into VCF - https://docs.vmware.com/en/VMware-Cloud-Foundation/5.1/vcf-admin/GUID-45A77DE0-A38D-4655-85E2-BB8969C6993F.html 
• HCI Mesh with VCF - https://docs.vmware.com/en/VMware-Cloud-Foundation/5.1/vcf-admin/GUID-1F86850D-E95E-40A8-AFC5-BE58D504D739.html
   

Enabling Two-Node and vSAN Stretched Clusters

Overview

To configure a vSAN two node cluster or stretched, select the appropriate option from the options given in [vSAN Cluster] > Configure > Services:

Note: A witness appliance is required for two node and stretched vSAN clusters.

For more information on vSAN two-node and stretched cluster, see the ‘vSAN Proof of Concept: vSAN Stretched Cluster and Two-Node Overview and Testing’ guide.

Enabling vSAN Stretched Clusters in VMware Cloud Foundation

VCF includes dedicated processes to automate the deployment and configuration of core infrastructure including vSAN services.   In fact, these processes are required and are the only supported methods within VCF. 

You can stretch a vSAN cluster in a workload domain across two availability zones within a region. Both availability zones must contain an equal number of hosts to ensure failover in case any of the availability zones goes down.

The default management vSphere cluster must be stretched before a VI workload domain cluster can be stretched. This ensures that the NSX control plane and management VMs (vCenter, NSX management cluster, and SDDC Manager) remain accessible if the stretched cluster in the primary availability zone goes down.

You cannot stretch a cluster in the following conditions:

• The cluster uses vSAN ESA
• The cluster has a vSAN remote datastore mounted on it
• The cluster shares a vSAN Storage Policy with any other clusters
• The cluster includes DPU-backed hosts
• The cluster is enabled for Workload Management (vSphere with Tanzu)

Review this link for more information: https://docs.vmware.com/en/VMware-Cloud-Foundation/5.1/vcf-admin/GUID-7B4CC729-20BD-4CC9-B855-B38F02F74D40.html

Check the vSAN Cluster Thoroughly

Once the vSAN network has been created and vSAN is enabled, you should check that each ESXi host in the vSAN cluster is able to communicate to all other ESXi hosts in the cluster. The easiest way to achieve this is via the vSAN Health Check.

Why Is This Important?

vSAN is dependent on the correct hardware and firmware combinations, as well as the network (configuration, reliability, performance, etc.). One of the most frequent causes of requesting support is either an incorrect network configuration or the network not performing as expected.

Use Health Check to Verify vSAN Functionality

Running individual commands from one host to all other hosts in the cluster can be tedious and time-consuming. vSAN has an integrated health check system. One of the first tasks to do after setting up any vSAN cluster is to perform a vSAN Health Check. 

To run a vSAN Health Check, navigate to [vSAN Cluster] > Monitor > vSAN > Skyline Health and click the RETEST button.

Note: The Skyline Health interface provides the option to see the health findings in both a tile and list views. In this case, the screenshots below, show a “Cluster Health Score” of 99% with no Unhealthy findings in both tile and list views.

If any of the health checks fail, select the appropriate finding, and then click on the Troubleshoot button for information on how to resolve the issue. This brings up a new screen providing details on the cause and recommended fix.  The screen also contains an Ask VMware button where appropriate, which will take you to a VMware Knowledge Base article detailing the issue, and how to troubleshoot and resolve it.

Ensure that the latest version of the HCL has been downloaded and run a RETEST on the Health check screen. Navigate to the vSAN HCL DB up-to-date health finding, expanding the finding, then click View Current Result. On the Result Screen select GET LATEST VERSION ONLINE.

If there is no internet connectivity, download the latest JSON from https://partnerweb.vmware.com/service/vsan/all.json (see https://kb.vmware.com/s/article/2145116 for more details) and select UPDATE FROM FILE...

The Performance Service is enabled by default. You can check its status from [vSAN Cluster] > Configure > vSAN > Services. If it needs to be manually enabled, click the EDIT button next to Performance Service and turn it on using the defaults. The Performance Service provides vSAN performance metrics to vCenter and other tools like Aria Operations.

To ensure everything in the cluster is optimal, the health service will also check the hardware against the VMware Compatibility Guide (VCG) for vSAN, verify that the networking is functional, and that there are no underlying disk problems or vSAN integrity issues.

Manually Checking against the VCG

The following commands are useful to help identify firmware and drivers in ESXi for comparison with the VCG. First, log in to an ESXi host via SSH, then run the following commands to obtain the information from the server:

See the controller details:

esxcli vsan debug controller list

List VID DID SVID SSID of a storage controller or network adapter:

vmkchdev -l | egrep 'vmnic|vmhba'

Show which NIC driver is loaded:

esxcli network nic list

Show which storage controller driver is loaded:

esxcli storage core adapter list

Display driver version information:

vmkload_mod -s <driver-name> | grep -i version

Display NVMe driver information:

esxcli system module get -m nvme_pcie

For NVMe device info (replace X with the appropriate value):

esxcli nvme device get -A vmhbaX | egrep "Serial|Model|Firmware"

vSAN Basics

Deploy your first Virtual Machine

In this section, a VM is deployed to the vSAN datastore using the ‘vSAN default storage policy’, which stipulates a simple RAID-1 mirror. 

Note: Due to the way data is structured in vSAN ESA, it recommended in most circumstances to define a RAID-5 policy for VMs. 

To examine the default policy settings, navigate to Menu > Shortcuts >VM Storage Policies.

From there, select vSAN Default Storage Policy. Look under the Rules tab to see the settings on the policy:

We will return to VM Storage Policies in more detail later, but when a VM is deployed with the default policy, it should have a mirror copy of the VM data created. This second copy of the VM data is placed on storage on a different host or fault domain to enable the VM to tolerate any single failure.

Also note that object space reservation is set to 'Thin provisioning', meaning that the object should be deployed as “thin”. After we have deployed the VM, we will verify that vSAN adheres to both capabilities.

One final item to check before we deploy the VM is the current free capacity on the vSAN datastore. This can be viewed from the [vSAN Cluster] > Monitor > vSAN > Capacity. In this example, it is 1.37 TB.

Make a note of the free capacity in your environment before continuing with the deploy VM exercise.

To deploy the VM, simply follow the steps provided in the wizard.

Select New Virtual Machine from the Actions Menu.

Select Create a new virtual machine.

Provide a name for the VM:

Select a compute resource (if DRS is enabled on the cluster, select the cluster itself, otherwise select one of the hosts):

Up to this point, the virtual machine deployment process is identical to all other virtual machine deployments. It is the next section that may be unfamiliar: this is where a policy for the virtual machine is chosen. 

As per the screenshot below, select change the VM storage policy to ‘vSAN Default Storage Policy’.

Once the policy has been chosen, datastores are split into those that are either compliant or non-compliant with the selected policy. As seen below, only the vSAN datastore can utilize the policy settings in the vSAN Default Storage Policy, so it is the only one that shows up as Compatible in the list of datastores.

The rest of the VM deployment steps in the wizard are quite straightforward, and simply entail selecting ESXi version compatibility (leave at default), a guest OS, and customize hardware (no changes needed).

Verifying Disk Layout of a VM on vSAN 

Physical placement of data on vSAN is defined by storage polices, which we will look at in more detail in another section. Here, we look at how the data is placed with vSAN default policy.

Once the VM is created, select the new VM in the inventory, navigate to the Configure tab, and then select Policies. There should be two objects shown, "VM home" and "Hard disk 1". Both should show a compliance status of Compliant meaning that vSAN was able to deploy these objects in accordance with the policy settings.

To verify this, navigate to the [vSAN Cluster] > Monitor > Virtual Objects. Once again, both the “VM home” and “Hard disk 1” should be displayed. Select the VM, followed by View Placement Details.

This should display a physical placement of RAID 1 configuration with two components, each component representing a mirrored copy of the virtual disk. It should also be noted that the components are located on different hosts or fault domains. This implies that the policy setting to tolerate 1 failure is being adhered to, as each host is an implicit fault domain. Further, fault domains can be explicitly defined: for instance, hosts within a single rack. Thus, data is resilient to failure of the entire rack. For details on how to create fault domains, review Managing Fault Domains in vSAN Clusters - https://docs.vmware.com/en/VMware-vSphere/8.0/vsan-administration/GUID-8491C4B0-6F94-4023-8C7A-FD7B40D0368D.html

Physical Placement: vSAN OSA

In a vSAN OSA cluster, the ‘witness’ component is used to maintain a quorum on a per-object basis. For more information, refer to the VMware vSAN Design Guide on core.vmware.com

Physical Placement: vSAN ESA

In vSAN ESA, physical placement is a little different. Data is written into two legs: writes are first ingested into a performance leg and then coalesced and written to a capacity leg. Whilst the distribution of data on the capacity leg reflects the storage policy setting (RAID 1 vs. RAID 5, etc.), the performance leg is always a RAID-1 mirror (and the FTT of the policy is followed). For this reason, RAID-5 performance in vSAN ESA is at least on-par with RAID-1 performance on vSAN OSA. Thus, it is recommended, for most workloads, to define a RAID-5 policy for VMs on vSAN ESA.

For more information on object placement in vSAN ESA, visit https://core.vmware.com/vsan-esa 

Below we see how the vSAN default policy (RAID-1, FTT-1) distributes the objects (physical disk placement can also be seen per VM, by selecting the VM and navigating to Monitor > vSAN > Physical disk placement). Also note that no witness components are needed (as opposed to vSAN OSA) as there are enough data objects to maintain quorum:  

Physical Space Requirements 

The “object space reservation” policy setting defines how much space is initially reserved on the vSAN datastore for a VM's objects. By default, it is set to "thin provisioning", implying that the VM’s storage objects are entirely “thin” and consume no unnecessary space. Note the free capacity in the vSAN datastore after deploying the VM, we see that the free capacity is very close to what it was before the VM was deployed, as displayed:

Because we have not installed anything in the VM (such as a guest OS) - it shows that only a tiny portion of the vSAN datastore has so far been used, verifying that the object space reservation setting of "Thin provisioning" is working correctly (observe that the "Virtual disks" and "VM home objects" consume less than 1GB in total, as highlighted in the "Used Capacity Breakdown" section).

Do not delete this VM as we will use it for other tests going forward.

Configuring Fault Domains

As mentioned above, a single host is a fault domain, i.e. data is separated such that the failure of a host does not lead to data loss. On failure, data can be rebuilt elsewhere. We can also group hosts so that data is further spread, and data is better protected.

Fault domains can be defined at cluster creation time (see the sections above). To define (or alter) fault domains, thereafter, navigate to [vSAN Cluster] > Configure > vSAN > Fault Domains.

Fault domains can then be created by either dragging and dropping into the ‘plus’ icon, or by ticking the appropriate hosts and selecting Move Hosts.

Creating a Snapshot

Using the virtual machine created previously, take a snapshot of it. The snapshot can be taken when the VM is powered on or powered off. The objectives are to see that:

• No setup is needed to make vSAN handle snapshots
• The process for creating a VM snapshot is unchanged with vSAN
• A successful snapshot delta object is created
• The policy settings of the delta object are inherited directly from the base disk object

From the VM object in vCenter, click Actions > Snapshots > Take Snapshot...

Take a Snapshot of the virtual machine created in the earlier step.

Provide a name for the snapshot and optional description.

Once the snapshot has been requested, monitor tasks and events to ensure that it has been successfully captured. Once the snapshot creation has completed, additional actions will become available in the snapshot drop-down window. For example, there is a new action to Revert to Latest Snapshot and another action to Manage Snapshots.

Choose the Manage Snapshots option. The following is displayed. It includes details regarding all snapshots in the chain, the ability to delete one or all of them, as well as the ability to revert to a particular snapshot.

To see snapshot delta object information from the UI, navigate to Monitor > vSAN > Physical disk placement.

There are now three objects that are associated with that virtual machine. First is the "VM Home" namespace. "Hard disk 1" is the base virtual disk, and "Hard disk 1 - poc-test-vm1.vmdk" is the snapshot delta. Notice the snapshot delta inherits its policy settings from the base disk that needs to adhere to the vSAN Default Storage Policy.

For vSAN ESA, another object is created (with the same layout as Hard disk 1):

The snapshot can now be deleted from the VM. Monitor the VM’s tasks and ensure that it deletes successfully.

Clone a Virtual Machine

We will continue to use the same VM as before. This time make sure the VM is powered on first. 

There are several different cloning operations available with vSAN. Here we will “Clone to Virtual Machine”. The cloning operation is a straightforward click-through operation. This next screen is the only one that requires human interaction. Simply provide the name for the newly cloned VM, and a folder if desired.

We are going to clone the VM in the vSAN cluster, so this must be selected as the compute resource.

On the “Select Storage” screen select the source datastore for the VM, “vsanDatastore”. This will all be pre-selected for you if the VM being cloned also resides on the vsanDatastore.

Select from the available options (leave unchecked - default)

This will take you to the “Ready to complete” screen. If everything is as expected, click FINISH to commence the clone operation. Monitor the VM tasks for the status of the clone operation.

This completes the cloning section of this guide. Do not delete the newly cloned VM, we will be using it in subsequent tests.

vMotion a Virtual Machine Between Hosts

The first step is to power-on the newly cloned virtual machine. We will migrate this VM from one vSAN host to another vSAN host using vMotion.

Note: Take a moment to revisit the network configuration and ensure that the vMotion network is distinct from the vSAN network. If these features share the same network, performance will not be optimal.

First, determine which ESXi host the VM currently resides on. Selecting the Summary tab of the VM shows this, in the ‘related objects window’.

Next, select ‘migrate’, either from the VM ‘actions’ menu or by right-clicking on the VM:

"Migrate" allows you to migrate to a different compute resource (host), a different datastore or both at the same time. In this initial test, we are simply migrating the VM to another host in the cluster, so this initial screen should be left at the default of “Change compute resource only”.

Select Change compute resource only:

Then select a destination host:

Select a destination network and click Next.

Leave the default (high) on the vMotion Priority window, click Next.

At the “Ready to complete” window, click on FINISH to initiate the migration. If the migration is successful, the summary tab of the virtual machine should show that the VM now resides on a different host.

Verify that the VM has been migrated to a new host:

This completes the “VM migration using vMotion” section of this guide. As you can see, vMotion works just great with vSAN. Do not delete the migrated VM: we will be using it in subsequent tests.

VMware vSphere® Storage vMotion® (Storage vMotion) VM Between Datastores

This test will only be possible if there is space on a VMFS datastore (such as the boot volume) on the host housing the VM or you have another datastore type available, such as an NFS share. The objective of this test is to successfully migrate a VM from another datastore type into vSAN and vice versa.

Mount an NFS Datastore to the Hosts (optional)

The steps to mount an NFS datastore to multiple ESXi hosts are described in the vSphere Administrators Guide. See the Create NFS Datastore in the vSphere Client topic for detailed steps.

Storage vMotion a VM from vSAN to another Datastore Type

Currently, the VM resides on the vSAN datastore. As we did before, launch the migrate wizard, however, on this occasion move the VM from the vSAN datastore to another datastore type by selecting Change storage only.

Select destination datastore and change the VM Storage Policy to Datastore Default as the vSAN policy will not apply to a VMFS (or NFS) datastore. Here we are migrating to the local VMFS datastore on the host:

On the “Ready to complete” screen, click FINISH to initiate the migration.

Once the migration completes, the ‘Datastores’ tab can be used to examine the datastore on which the VM resides.

Verify that the VM has been moved to the new storage.

VM Storage Policies and vSAN

VM Storage Policies form the basis of VMware’s Software-Defined Storage vision. A VM Storage Policy dictates how vSAN should place data (as well as some other features) across the physical resources, such as RAID type, number of stripes and compression (for vSAN ESA). Previously, we have deployed our VMs onto the ‘vSAN Default Storage Policy’.

The actual status of the data may not reflect the policy definition (for instance, if there is a failure, or if the policy has recently been changed). Thus, VM disks can be either compliant or non-compliant with the assigned storage policy. The latter is usually a temporary state, until the system stabilizes. Once the rebuild (or other operation) is complete, compliance is automatically regained.

Note: Storage policies are applied per VMDK in vSAN OSA and per VM in vSAN ESA. Further, the recommended storage policy for vSAN ESA clusters is RAID-5 (see the vSAN features guide for more information). 

Create a New VM Storage Policy

We will build a policy with RAID 1 and a stripe width of two. The VM Storage Policies can be accessed from the 'Shortcuts' page on the vSphere client, as shown below.

Here we see the existing policies already in place, such as the ‘vSAN Default Storage Policy’ (already used to deploy VMs in the ‘Basic vSphere Functionality’ section of this guide). 

To create a new policy, click on Create.

The next step is to provide a name and an optional description for the new VM Storage Policy. Since this policy will contain a stripe width of two, we have given it a name to reflect this. You may also give it a name to reflect that it is a vSAN policy.

The next section sets the policy structure. We select Enable rules for "vSAN" Storage to set a vSAN specific policy:

Now we get to the point where we create a set of rules. The first step is to select the availability of the objects associated with this rule. Set the failures to tolerate to one failure (RAID 1)

We then set the Advanced Policy Rules. Once this is selected, the six customizable capabilities associated with vSAN are exposed.  Since this VM Storage Policy is going to have a requirement where the stripe width of an object is set to two, this is what we select from the list of rules. It is officially called “Number of disk stripes per object”. 

Note: The general recommendation is to keep the number of stripes at the default, one (unless troubleshooting performance or for specific scenarios). Here, we are using this setting to clearly demonstrate how a storage policy affects storage components.

The next screen shows the datastores that are compatible with the policy. In this case, only the vsanDatastore is compatible with the policy settings.

Note: This does not mean that the vSAN datastore can successfully deploy a VM with this policy. It simply means that the vSAN datastore understands the rules or requirements in the policy. The ‘force provisioning’ option will try and apply the policy without first checking if it can be fulfilled by the cluster.

Review the settings on the next screen and click FINISH to create the policy:

We can now go ahead and deploy a VM with this new policy and see what effect it has on the layout of the underlying storage objects.

Note: vSAN includes some pre-defined storage policies for the vSAN File Service, named ‘FSVM_Profile_DO_NOT_MODIFY’. These policies are used internally by vSAN for vSAN file services and should not be modified.

Deploy a new VM with a New Storage Policy

The workflow to deploy a New VM remains the same until we get to the point where the VM Storage Policy is chosen. This time, instead of selecting the default policy, select the newly created policy, as shown below. As before, the vsanDatastore should show up as the compatible datastore, and thus the one to which this VM should be provisioned. To illustrate clearly, we will see how this works on a vSAN OSA cluster to begin with.

Now let's examine the layout of this virtual machine, and see if the policy requirements are met, i.e. do the storage objects of this VM have a stripe width of 2. First, ensure that the VM is compliant with the policy by navigating to [VM] > Configure > Policies, as shown here:

The next step is to select the [vSAN Cluster] > Monitor > vSAN > Virtual Objects and check the layout of the VM’s storage objects. Select the "Hard Disk 1" object and click on the View Placement Details:

Here, in our vSAN OSA cluster, we can see the two (RAID 0) stripe components, as defined by the policy. Each striped component must be placed on its own physical (capacity) disk. There are enough physical disks to meet this requirement here. However, a request for a larger stripe width would not be possible in this configuration. The stripes are across physical disks, and not necessarily disk groups. 

However, on examining the “VM Home” object, we see an apparent policy violation – there is no striping seen. This is by design, as there is no performance gain by striping this object, so the policy setting is ignored.

It should also be noted that snapshots taken of this base disk continue to inherit the policy of the base disk, implying that the snapshot delta objects will also be striped.

With a vSAN ESA cluster, the “Hard Disk 1” object is striped, as per the policy on the capacity leg.

Again, the “VM home” object ignores the stripe width from the policy, and the capacity leg is still only RAID 1, single stripe:

Note: The stripe width setting may have unintended consequences if used on an ESA cluster. For more information, visit:
https://core.vmware.com/blog/stripe-width-storage-policy-rule-vsan-esa

Assign a new Storage Policy to an Existing VM

You can choose to modify the VM Storage Policy mapping of an existing VM deployed on the vSAN datastore. The configuration of the objects associated with the VM will be modified to comply with the newer policy. For example, if the number of failures to tolerate (FTT) is increased, newer components would be created, synchronized with the existing object, and subsequently, the original object is discarded. VM Storage policies can also be applied to individual objects.

Here, we will illustrate this (on a vSAN ESA cluster) by applying a new policy to the VM, increasing the failures to tolerate to FTT=2 (keeping RAID 1 and the stripes=2). Once again, we create a new policy. Below, we have named the policy “R1 FTT=2 Stripe Width=2’. 

First, we set the FTT value:

Then, as before, the stripe width:

Then, we navigate to our VM, then Configure > Policies and click on EDIT VM STORAGE POLICES

This takes you to the edit screen, where the policy can be changed. The new policy can then be selected from the drop-down list

Once the policy is selected, click the OK button as shown above to ensure the policy gets applied to all storage objects. The VM Storage Policy should now appear updated for all objects. Now when you revisit the Configure > Policies view, you should see the changes in the process of taking effect (Reconfiguring) or completed.

Looking at the physical disk placement, the capacity leg now has three sets of stripes (as expected). Moreover, as we have increased the FTT, the performance leg now has an extra stripe set:

Modify a VM Storage Policy

The workflow above is useful when you only need to modify the policy of one or two VMs, but if you need to change the VM Storage Policy of a significant number of VMs then this can be a little onerous. Instead, we can update by simply changing the policy used by those VMs. All VMs using those policies can then be “brought to compliance” by reconfiguring their storage object layout to make them compliant with the policy. We shall look at this next.

Note: Modifying or applying a new VM Storage Policy leads to additional backend IO as the objects are being synchronized.

In this task, we shall modify an existing VM Storage policy to set the ‘Object Space Reservation’ parameter to 25%. This means that each storage object will now reserve 25% of the VMDK size on the vSAN datastore. Since all VMs were deployed with 40GB VMDKs with Failures to tolerate=1 failure - RAID-1 (Mirroring), the reservation value will be 20 GB.

As the first step, note the amount of free space in the vSAN datastore. This would help ascertain the impact of the change in the policy.

Select StripeWidth=2 policy from the list of available policies, and then the Edit Settings option. Navigate to vSAN > Advanced Policy Rules and modify the Object space reservation setting to 25%, as shown below:

Proceed to complete the wizard with default values and click FINISH. A pop-up message requiring user input appears with details of the number of VMs using the policy being modified. This is to ascertain the impact of the policy change. Typically, such changes are recommended to be performed during a maintenance window. You can choose to enforce a policy change immediately or defer it to be changed manually at a later point. Leave it at the default, which is “Manually later”, by clicking Yes as shown below:

Next, select the Storage policy -StripeWidth=2 and click on the VM Compliance tab in the bottom pane. It will display the two VMs along with their storage objects, and the fact that they are no longer compliant with the policy. They are in an “Out of Date” compliance state as the policy has now been changed.

You can now enforce a policy change by navigating to [VM Storage Policies] and clicking on Reapply VM Storage Policy:

When this button is clicked, the following popup appears.

When the reconfigure activity completes against the storage objects, and the compliance state is once again checked, everything should show as Compliant.

Since we have now included an ObjectSpaceReservation value in the policy, you may notice corresponding capacity reduction from the vSAN datastore.

For example, the two VMs with the new policy change have 40GB storage objects. Therefore, there is a 25% ObjectSpaceReservation implying 10GB is reserved per VMDK. So that's 10GB per VMDK, 1 VMDK per VM, 2 VMs equals 20 GB reserved space, right? However, since the VMDK is also mirrored, there is a total of 40GB reserved on the vSAN datastore.

IOPS Limits

vSAN incorporates a quality-of-service feature that can limit the number of IOPS an object may consume. IOPS limits are enabled and applied via a policy setting. The setting can be used to ensure that a particular virtual machine does not consume more than its fair share of resources or negatively impact the performance of the cluster.

Here, we demonstrate setting IOPS limits by creating a new policy (as above). We can then set the IOPS limit by navigating to the ‘Advanced Policy Rules’ tab. In our example, we have set the IOPS limit to 1000:

The IOPS limit value is calculated as the number of 32KB IOs per second. Therefore, in this example, where we have a value of 1000, the IOPS limit is 1000x32KB=32MB/s. If I/O against the VM or VMDK should rise above the threshold, the additional I/O will be throttled. Note that any I/O incurred by a snapshot is counted too.

Space Efficiency & Encryption Defined by Policy (vSAN ESA)

In vSAN ESA compression can be enabled/disabled by VM storage policy. The policy applies to any new data, so turning off compression will only affect any new writes. Note that compression is turned on by default in vSAN ESA. 

In the example below, we create a new policy (as per above) and then navigate to the ‘Storage rules’ tab to configure the services. Here, we enable encryption and disable compression:

Note: Turning off compression will affect new writes only. Existing data is not affected.

APPENDIX A: vSAN RDMA Configuration & Troubleshooting

Overview

RDMA (Remote Direct Memory Access) provides a more efficient network transport layer than traditional TCP connections. For more details visit https://core.vmware.com/blog/vsan-7-update-2-rdma-support

Requirements:

• A supported physical network adapter that has RDMA RoCEv2 capabilities; see the vSAN VCG
• A switch that supports RDMA RoCEv2 and associated configuration

Note that neither vSAN Stretched Clustering nor two-node clusters are supported with RDMA and that LACP should not be configured on the network uplinks.

To enable RDMA support, it can be achieved using HCI Quick Start wizard 

Toggle to enable and select next

Alternatively on an existing cluster, toggle to enable and click on Apply:

Configuration Example

First, enable PFC support on the switch. The example below shows the configuration steps on a Mellanox SN2100:

Enable PFC:

switch01 [standalone: master] (config) # dcb priority-flow-control enable
This action might cause traffic loss while shutting down a port with priority-flow-control mode on
Type 'yes' to confirm enable pfc globally: yes
 
Enable Switch PFC for priority 4
dcb priority-flow-control priority 4 enable

Assign PFC to port (ESXi uplink):

switch01 [standalone: master] (config) # interface ethernet 1/9 dcb priority-flow-control mode on force

Verify RDMA available adapter through ESXi shell:

[root@localhost:~] esxcli rdma device list

 Name     Driver      State    MTU  Speed     Paired Uplink  Description
-------  ----------  ------  ----  --------  -------------  -----------
vmrdma0  nmlx5_rdma  Active  4096  100 Gbps  vmnic4         MT27700 Family  
vmrdma1  nmlx5_rdma  Active  4096  100 Gbps  vmnic5         MT27700 Family  

[root@localhost:~]esxcli network nic dcb status get -n vmnic4

Nic Name: vmnic4
Mode: 3 - IEEE Mode
Enabled: true
Capabilities:
Priority Group: true
Priority Flow Control: true
PG Traffic Classes: 8
PFC Traffic Classes: 8
PFC Enabled: true
PFC Configuration: 0 0 0 0 1 1 0 0

[root@localhost:~] esxcli rdma device protocol list
 
Device    RoCE v1   RoCE v2   iWARP
-------   -------   -------   -----
vmrdma0  true      true  false
vmrdma1  true      true  false

RDMA troubleshooting

First, verify that the physical network adapters support RDMA. On each host, navigate to Configure > Networking > Physical adapters > [adapter] > RDMA:

Verify if RDMA adapters are bound to VMkernel interface

On each host, navigate to Configure > Networking > RDMA adapters > [vmrdma#] > Bound VMkernel Adapter:

Note: The RDMA support flag is required for the final setup to enable vSAN RDMA transport (if the RDMA flag is not visible, then double check the hardware specification of the adapter along with driver/firmware versions and the vSphere HCL).

vSphere vSAN RDMA uses RoCEV2 as its protocol layer. When there is no RDMA support available on the physical link or setup, communication falls back to standard legacy TCP/IP automatically.

Esxtop provides additional fields for enablement through the ‘f’ key:

* A:  NAME =  Name of device
B:  DRIVER =  driver
C:  STATE =    State
* D:  TEAM-PNIC = Team Uplink Physical NIC Name
* E:  PKTTX/s =  Packets Tx/s
* F:  MbTX/s =  Megabits Tx/s
* G:  PKTRX/s =  Packets Rx/s
* H:  MbRX/s =  Megabits Rx/s
I:  %PKTDTX =  % Packets Dropped (Tx)
J:  %PKTDRX =  % Packets Dropped (Rx)
* K:  QP =  Number of Queue Pairs Allocated
L:  CQ =  Number of Completion Queue Pairs Allocated
M:  SRQ =  Number of Shared Receive Queues Allocated
* N:  MR =  Memory Regions Allocated

Toggle fields with a-n, any other key to return

Default setup enables only the minimum requirement for performance for MB/s, queue pairs (QP) and allocated memory regions verbs (MR). For in-depth RDMA functionality, please contact your hardware vendor.

Run the following to obtain detailed adapter statistics. Check for any errors here. Queue pairs are adjusted automatically by requirement:

[root@localhost:~] esxcli rdma device stats get -d vmrdma0
   Packets received: 1576258135
   Packets sent: 899769661
   Bytes received: 40653333761546
   Bytes sent: 1079424621290
   Error packets received: 0
   Error packets sent: 0
   Error length packets received: 0

APPENDIX B: Cleanly Removing vSAN Configuration

vCLS Retreat Mode

On occasion, it may become necessary to remove a vSAN cluster and reset hosts to a ‘clean’ state.

To expedite the process, it is advisable to first put vCLS into retreat mode. This will delete the vCLS VMs and make it easier to remove the vSAN datastore and put hosts into maintenance mode, etc.

To achieve this, an vCenter advanced setting, ‘config.vcls.clusters.[domain].enabled’ needs to be set.

The procedure to do this is detailed in the documentation here: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-F98C3C93-875D-4570-852B-37A38878CE0F.html

To make this easier a script is available here to use (download to a Linux or Mac host, uses govc):
https://github.com/vmware-tanzu-experiments/vsphere-with-tanzu-proof-of-concept-samples/blob/main/VCF/vCLS.sh

Remove vSAN Partitions and Clear Data

The next step is to turn off vSAN from vCenter, under [cluster] > Configure > Services > vSAN. If for some reason this step encounters errors, the method below may be useful.

First, open an SSH session to all hosts in the cluster and list the disks used by vSAN by using the command:

vdq -iH

The next step depends on the type of cluster

OSA Clusters

Remove the cache device from each disk group, using the command:

esxcli vsan storage remove -s [cache device]

ESA Clusters

Remove disks from the storage pool, using the command:

esxcli vsan storagepool remove -d [device]

Next, relabel the disks:

partedUtil mklabel /vmfs/devices/disks/[disk] gpt

Again, to make this easier, a script is available to help with this:

OSA: https://github.com/vmware-tanzu-experiments/vsphere-with-tanzu-proof-of-concept-samples/blob/main/VCF/vsan-remove-esa.sh

ESA: https://github.com/vmware-tanzu-experiments/vsphere-with-tanzu-proof-of-concept-samples/blob/main/VCF/vsan-remove-esa.sh