vSAN Proof of Concept: vSAN Architecture Overview & Setup
What’s New in vSAN 8.0
vSAN 8.0 is a major milestone release that includes a new architecture option for new vSAN clusters, and there are now two distinct ways to configure a vSAN cluster. New in vSAN 8.0 is the Express Storage Architecture (ESA), which optimizes the use of certified NVMe drives in a single tier, with a dynamic approach to data placement. The classic Original Storage Architecture (OSA), which is updated from vSAN 7, remains an option.
For a comprehensive overview of ESA, visit: https://core.vmware.com/vsan-esa
Updates to the OSA include:
- Change in buffer device capacity, increasing from 600GB to 1.6TB.
- For HCI Mesh, the number of server and client clusters increases from 5 to 10
- Improvements in vSAN File Services operations
- vSAN Boot time optimizations
Introduction
Hardware Selection
Choosing the appropriate hardware is one of the most important factors in the successful validation of vSAN.
There are many variables with hardware (drivers, controller firmware versions) so be sure to choose hardware that is on the VMware Compatibility List. See the ‘Prerequisites’ section of this guide for more information.
General approach to Testing vSAN
Once the appropriate hardware is selected for testing, it is useful to define the use case, goals, expected results and success metrics.
The testing lifecycle can be broken into three phases:
Day-0
The post-design phase, installation of the infrastructure components, including: the hypervisor (in the case of ESXi); control plane (vCenter); physical network uplinks and upstream network devices; essential services, such as DNS and NTP; VLANs, etc.
Day-1
Setup and configuration of the required solution (in the case of vSAN)
Day-2
Operational aspects and monitoring
The most important aspects to validate when testing are:
- Successful vSAN configuration and deployment
- VMs successfully deployed to vSAN Datastore
- Reliability: VMs and data remain available in the event of failure (host, disk, network, power)
- Serviceability: Maintenance of hosts, disk groups, disks, clusters
- Performance: vSAN and selected hardware can meet the application, as well as business needs
- Validation: vSAN features are working as expected (File Service, Deduplication and Compression, RAID-5/6, checksum, encryption)
- Day-2 Operations: Monitoring, management, troubleshooting, and upgrades
These can be grouped into three common types: resiliency testing, performance testing, and operational testing.
Resiliency Testing
As with any storage solution, failures can occur on hardware components at any time due to age, temperature, firmware, etc. Such failures can occur among storage controllers, disks, nodes, and network devices among other components. As a software solution, vSAN is designed to be resilient against these failures. In this guide, we will examine how to systematically test against disk, host, network, and control plane failures.
Operational Testing
Understanding how the solution behaves during normal (or “day two”) operations is important to consider as part of the evaluation. Fortunately, because vSAN is embedded within the ESXi hypervisor, many vSAN operations tasks are simply extensions of normal vSphere operations. Adding hosts, migrating VMs between nodes, and cluster creation are some of the many operations that are consistent between vSphere and vSAN.
Performance Testing
Before embarking on testing, it is important to set clear objectives and understand the performance requirements of the environment. Close attention to details such as workload I/O profiles, latency and hotspots is needed. In this guide, we will explore how to conduct performance tests with a consistent, methodological approach.
Prerequisites
Hardware Compatibility
Plan on testing a reasonable hardware configuration resembling a production-ready environment that suits your business requirements. Refer to the VMware vSAN Design and Sizing Guide for information on design configurations and considerations when deploying vSAN.
As vSAN is a software solution, it is critical to ensure that well supported, enterprise class hardware components are used. The VMware Compatibility Guide (or “VCG”) lists components that have been tested and certified for use with vSAN. In addition, the vSAN Hardware Compatibility List (or “HCL”) lists hardware compatibility specific to vSAN. Note that the terms “VCG” and “HCL” may be used interchangeably (both within this guide, and in other documentation), but essentially pertain to hardware compatibility.
Note: if using a vSAN Ready Node or appliance, the factory-installed hardware is guaranteed to be compatible with vSAN. However, be mindful that BIOS updates and firmware and device driver versions may be out of date and should be checked
Below we explore how to use the compatibility guide for vSAN:
vSAN ESA Ready Nodes
Navigate to https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsanesa
Ensure “vSAN ESA” is selected from “What are you looking for”:
For further vSAN ESA hardware guidance, see:
https://www.vmware.com/resources/compatibility/vsanesa_profile.php
https://kb.vmware.com/s/article/90343
BIOS
Navigate to http://www.vmware.com/resources/compatibility/search.php
Ensure "Systems / Servers" is selected from "What are you looking for":
Network cards
Navigate to https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io
Ensure "IO Devices" is selected from "What are you looking for"
Select "Network" from "I/O Device Type" field:
vSAN Storage I/O controllers & disks
Navigate to https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan
Ensure "vSAN" is selected from "What are you looking for"
Scroll to 'STEP 3' and look for the link to "Build Your Own based on Certified Components":
Then, from the ‘Build Your Own’ page, choose the appropriate device (e.g., I/O controller, SDD or HDD) to search for:
vSAN Setup Assumptions
The list below details the setup requirements prior to using this guide:
- Hosts with supported hardware and firmware (see above)
- All hosts with ESXi 8 (build number 20513097) or newer deployed
- vCenter Server 8 (build number 20519528) or newer has been deployed to manage these ESXi hosts
- DNS and NTP services are available
- The hosts should not be added to any cluster
- Unless using the Quickstart workflow, each host should have a management network VMkernel and a vMotion network VMkernel ports already configured.
- A set of IP addresses, one per ESXi host will be needed for the vSAN network VMkernel ports. The recommendation is that these are all on the same VLAN and IPv4 or IPv6 network segment.
- If possible, configure internet connectivity for vCenter such that the HCL database can be updated automatically. (Internet connectivity is also a requirement to enable Customer Experience Improvement Program or CEIP).
- Optionally, for the purposes of testing Storage vMotion operations, an additional datastore type (such as NFS or VMFS) should be presented to all hosts
vSAN ESA or OSA
With vSAN 8.0 there are now two options for vSAN deployments. The proven, time-tested Original Storage Architecture – or OSA and the newer Extended Storage Architecture – or ESA optimized for the latest flash devices.
If using certified Ready Nodes or vSAN appliances with specific certified devices, then vSAN Express Storage Architecture (which has an optimized data path and placement for NVMe devices) is an option — note that at least 25Gbps networking is required. For all other configurations, the updated vSAN Original Storage Architecture is available.
The table below summarizes the minimum requirements for each architecture
|
vSAN 8.0 ESA |
vSAN 8.0 OSA |
Storage Device Minimums |
4 devices per host |
1 capacity device + 1 cache device per host |
Hardware Support |
ReadyNodes/Appliances Only
Only vSAN ESA certified NVMe devices |
ReadyNodes/Appliances or build your own with certified devices
Any SATA, SAS. NVMe certified device |
Networking Requirements (minimum) |
25Gbps |
10Gbps |
For guidance on vSAN ESA Ready Nodes, visit: https://www.vmware.com/resources/compatibility/vsanesa_profile.php
vSAN OSA All-Flash or Hybrid
There are several factors to consider if you plan to deploy vSAN OSA in an All-Flash vSAN configuration:
- All-Flash vSAN requires at least a 10Gb Ethernet network
- Flash devices are used for both cache and capacity
- Deduplication and Compression are space-efficiency features available in all-flash configuration and not available with hybrid configuration
- Erasure Coding (RAID 5/ RAID 6) is a space efficiency feature available on all-flash configuration only
- Flash read cache reservation is not used with all-flash configurations; reads come directly from the capacity tier SSDs
- Endurance and performance classes now become important considerations for both cache and capacity layers
Enabling vSAN
Using Quickstart
Either follow this section to configure the cluster or use the next two sections to do so manually.
Note that if you deployed vSAN using the bootstrap feature of vCenter deployment, you will not be able to use Quickstart to configure your environment.
When creating a new cluster, we are presented with a dialog to enable services. Provide a name for the cluster and select vSAN from the list of services. We are also able to enable vSAN ESA (the default). Note that once the cluster is created with the ESA flag, it cannot be changed unless the cluster is re-created.
We can also setup the cluster to use a single image (thereby enabling vLCM). For more information on vLCM, see: https://core.vmware.com/resource/introducing-vsphere-lifecycle-management-vlcm
Once the cluster has been created, navigate to Configure > Quickstart:
The next step is to add hosts. Clicking on the 'Add' button on the 'Add hosts' section presents the dialog below. Multiple hosts can be added at the same time (by IP or FDQN). Additionally, if the credentials of every host are the same, tick the checkbox above the list to quickly complete the form. If the hosts are already in vCenter, they will appear in the ‘Existing hosts’ tab.
Note: it may be useful to leave one host out of the configuration here to later demonstrate cluster expansion.
Once the host details have been entered, click next. You are then presented with a dialog showing the thumbprints of the hosts. If these are as expected, tick the checkbox(es) and then click OK:
A summary will be displayed, showing the vSphere version on each host and other details. Verify the details are correct:
On the next screen, we have the option to import the ESXi an image from a host (to set as the cluster’s new image). Select an option and continue
Finally, review and click Finish if everything is in order:
After the hosts have been added, validation will be performed automatically on the cluster. Check for any errors and inconsistencies and re-validate if necessary:
The final step is to configure the cluster. After clicking on 'Configure' on step 3, the following dialog allows for the configuration of the distributed switch(es) for the cluster. Leave the default 'Number of distributed switches' set to 1 and assign a name to the switch and port groups:
Scroll down and configure the port groups and physical adapters as needed:
On the next two screens, set the VLAN and IP addresses to be used for the vMotion and vSAN for each host. The ‘autofill’ function can be used to input consecutive IP addresses
Select the type of deployment: standard or stretched cluster. Enable any extra features, such as deduplication and compression. Check everything is correct.
Note: here we select ‘Single site cluster’ as the deployment type. For stretched clusters, refer to the stretched cluster guide.
Next, ensure that an NTP server is added to the configuration (under ‘Host Options’). As vSAN is a distributed system, features may not work as expected if there is a time drift between servers (ideally, also ensure that vCenter has the same time source)
On the next screen, select the disks to use. For a vSAN OSA cluster, the system will attempt to select disks automatically as cache or capacity (on a vSAN ESA cluster, there is no tier selection):
Finally, check everything is as expected and click finish:
Monitor the creation in the task view and wait for the cluster to be formed.
Using Quickstart, we have created a cluster with vSAN enabled and the correct networking in place. The next two sections (vSAN Network Setup and enabling vSAN Services on a Cluster) can be skipped.
vSAN Network Setup
Note: if Quickstart was used (as per the earlier section) then this section can be skipped.
All ESXi hosts in a vSAN cluster communicate over a vSAN network. For network design and configuration best practices please refer to the VMware vSAN Design Guide on https://core.vmware.com/resource/vmware-vsan-design-guide
The following example demonstrates how to configure vSAN networking services on an ESXi host.
Note: it may be useful to leave one host out of the configuration here to later demonstrate cluster expansion.
Creating a VMkernel Port for vSAN
In many deployments, vSAN may be sharing the same uplinks as the vSphere management and vMotion traffic, especially when 10GbE or faster NICs are utilized. When sharing vSAN traffic with other traffic types, VMware recommends leveraging quality of service controls on virtual distributed switches (vDS) to this end. Note that licensing for distributed virtual switches is included with all versions of vSAN.
To create a vSAN VMkernel port, follow these steps:
First, we select an ESXi host in the inventory. Then navigate to Configure > Networking > VMkernel Adapters. Click on the icon for Add Networking, as highlighted below:
Ensure that VMkernel Network Adapter is chosen as the connection type:
Next, we can choose either an existing standard switch, vDS portgroup or create a new standard switch. In this example, we have chosen ‘vSwitch0’:
After the target switch or network is chosen, we then set up the port properties. Select ‘vSAN’ to enable vSAN services on this vmkernel port, as well as any VLAN ID:
Next, we apply an IP address to the vmkernel port:
Finally, check the input for any errors and click ‘Finish’ to create the vmkernel port.
If the creation of the VMkernel port was successful, it will appear in the list of VMkernel adapters.
This configuration process must then be repeated for all other ESXi hosts that will participate in the vSAN cluster (including the host outside the cluster that will be added later).
Enabling vSAN Services on a Cluster
Note: if Quickstart was used (as per the earlier section) then this section can be skipped.
Once all the pre-requisites have been met, vSAN can be configured. To enable vSAN complete the following steps:
Select the cluster on which you wish to enable vSAN then navigate to Configure > vSAN > Services
To set up vSAN for this cluster, select ‘I need a local vSAN datastore’. The other option relates to HCI Mesh.
Next, select which setup is required. Typically, the “Standard vSAN cluster” is selected here. Click on ‘Configure’. This will open a wizard to configure vSAN:
The first screen is a compatibility check for vSAN ESA. If the tests are successful, then the toggle to enable vSAN ESA is enabled. Here we can decide to use vSAN ESA or OSA: the next two subsections describe the setup for each.
vSAN OSA Setup
Toggle the selector to disable vSAN ESA. We can then choose which services to enable:
Note: The process of later enabling deduplication and compression or encryption of data at rest can take quite some time, depending on the amount of data that needs to be migrated during the rolling reformat. In a production environment, if deduplication and compression is a known requirement, it is advisable to enable this while enabling vSAN to avoid multiple occurrences of rolling reformat.
In the next screen, you can claim all the disks of the same type for either vSAN caching tier or capacity tier. For each listed disk make sure it is listed correctly as a Flash/HDD, and Caching/Capacity drive:
Optionally, create any fault domains (unless directed otherwise, vSAN places data components across hosts, so that a single host failure does cause data loss. This concept can be expanded across several hosts or fault domains. See later section on ‘Configuring Fault Domains’ for more information):
Review the details and click finish to create the cluster.
vSAN ESA Setup
If vSAN ESA is required, the next screen shows the available services.
We can choose which services to enable (note that compression is defined by VM storage policy).
Note: Data-At-Rest encryption needs to be selected before claiming disks. This cannot be modified after the disks have been added for vSAN.
On the next screen select the disks to claim for the storage pool:
Then optionally create any fault domains:
Review the details and click finish to create the cluster.
Two Node and vSAN Stretched Clusters
To configure a vSAN two node cluster or stretched, select the appropriate option from the options given in [Cluster] > Configure > Services:
Note A witness appliance is required for two node and stretched vSAN clusters.
For more information on vSAN two-node and stretched cluster, see the ‘vSAN Stretched Cluster and Two-Node Overview and Testing’ guide.
vSAN RDMA
RDMA (Remote Direct Memory Access) provides a more efficient network transport layer than traditional TCP connections. RDMA and specifically RoCEv2 support for vSAN was introduced in vSphere 7.0U2. For more details visit https://core.vmware.com/blog/vsan-7-update-2-rdma-support
Requirements:
- A supported physical network adapter that has RDMA RoCEv2 capabilities; see the vSAN VCG
- A switch that supports RDMA RoCEv2 and associated configuration.
Note that vSAN Stretch Cluster or two node is not supported with RDMA and that LACP should not be configured on the network uplinks
To enable RDMA support, navigate to [vSAN Cluster] > Configure > Services > Network > Edit:
Toggle to enable and click on apply:
For more details on RDMA, including troubleshooting steps, see Appendix A.
Check the vSAN Cluster Thoroughly
Once the vSAN network has been created and vSAN is enabled, you should check that each ESXi host in the vSAN cluster is able to communicate to all other ESXi hosts in the cluster. The easiest way to achieve this is via the vSAN Health Check.
Why Is This Important?
vSAN is dependent on the correct hardware and firmware combinations, as well as the network (configuration, reliability, performance, etc.). One of the most frequent causes of requesting support is either an incorrect network configuration or the network not performing as expected.
Use Health Check to Verify vSAN Functionality
Running individual commands from one host to all other hosts in the cluster can be tedious and time-consuming. Fortunately, since vSAN 6.0, vSAN has an integrated health check system. One of the first tasks to do after setting up any vSAN cluster is to perform a vSAN Health Check.
To run a vSAN Health Check, navigate to [vSAN cluster] > Monitor > vSAN > Skyline Health and click the RETEST button.
In the screenshot below, one can see that each of the health checks for networking has successfully passed, and all the hosts are in the same network partition:
If any of the network health checks fail, select the appropriate check, and examine the details pane on the right for information on how to resolve the issue. Each detailed view under the Info tab also contains an ‘Ask VMware’ button where appropriate, which will take you to a VMware Knowledge Base article detailing the issue, and how to troubleshoot and resolve it.
Ensure that the latest version of the HCL has been downloaded and run a RETEST on the Health check screen. Do this by selecting vSAN HCL DB up-to-date health check under the "Hardware compatibility" category, and choosing GET LATEST VERSION ONLINE.
If there is no internet connectivity, download the latest JSON from https://partnerweb.vmware.com/service/vsan/all.json (see https://kb.vmware.com/s/article/2145116 for more details) and select UPDATE FROM FILE...
The Performance Service is enabled by default. You can check its status from [vSAN cluster] > Configure > vSAN > Services. If it needs to be manually enabled, click the EDIT button next to Performance Service and turn it on using the defaults. The Performance Service provides vSAN performance metrics to vCenter and other tools like vRealize Operations Manager.
To ensure everything in the cluster is optimal, the health service will also check the hardware against the VMware Compatibility Guide (VCG) for vSAN, verify that the networking is functional, and that there are no underlying disk problems or vSAN integrity issues.
Manually Checking against the VCG
The following commands are useful to help identify firmware and drivers in ESXi for comparison with the VCG. First, log in to an ESXi host via SSH, then run the following commands to obtain the information from the server:
See the controller details:
esxcli vsan debug controller list
List VID DID SVID SSID of a storage controller or network adapter:
vmkchdev -l | egrep 'vmnic|vmhba'
Show which NIC driver is loaded:
esxcli network nic list
Show which storage controller driver is loaded:
esxcli storage core adapter list
Display driver version information:
vmkload_mod -s <driver-name> | grep -i version
Display NVMe driver information:
esxcli system module get -m nvme_pcie
For NVMe device info (replace X with the appropriate value):
esxcli nvme device get -A vmhbaX | egrep "Serial|Model|Firmware"
vSAN Basics
Deploy your first Virtual Machine
In this section, a VM is deployed to the vSAN datastore using the ‘vSAN default storage policy’, which is stipulates a simple RAID-1 mirror.
Note: for vSAN ESA, due to the way data is structured, it recommended in most circumstances to define a RAID-5 policy for VMs.
To examine the default policy settings, navigate to Menu > Shortcuts > VM Storage Policies.
From there, select vSAN Default Storage Policy. Look under the Rules tab to see the settings on the policy:
We will return to VM Storage Policies in more detail later, but when a VM is deployed with the default policy, it should have a mirror copy of the VM data created. This second copy of the VM data is placed on storage on a different host or fault domain to enable the VM to tolerate any single failure.
Also note that object space reservation is set to 'Thin provisioning', meaning that the object should be deployed as “thin”. After we have deployed the VM, we will verify that vSAN adheres to both capabilities.
One final item to check before we deploy the VM is the current free capacity on the vSAN datastore. This can be viewed from the [vSAN cluster] > Monitor > vSAN > Capacity. In this example, it is 1.37 TB.
Make a note of the free capacity in your environment before continuing with the deploy VM exercise.
To deploy the VM, simply follow the steps provided in the wizard.
Select New Virtual Machine from the Actions Menu.
Select Create a new virtual machine.
Provide a name for the VM:
Select a compute resource (if DRS is enabled on the cluster, select the cluster itself, otherwise select one of the hosts):
Up to this point, the virtual machine deployment process is identical to all other virtual machine deployments. It is the next section that may be unfamiliar: this is where a policy for the virtual machine is chosen.
As per the screenshot below, select change the VM storage policy to ‘vSAN Default Storage Policy’.
Once the policy has been chosen, datastores are split into those that are either compliant or non-compliant with the selected policy. As seen below, only the vSAN datastore can utilize the policy settings in the vSAN Default Storage Policy, so it is the only one that shows up as Compatible in the list of datastores.
The rest of the VM deployment steps in the wizard are quite straightforward, and simply entail selecting ESXi version compatibility (leave at default), a guest OS, and customize hardware (no changes needed).
Note: for vSAN ESA, due to the way data is structured, it recommended in most circumstances to define a RAID-5 policy for VMs.
Verifying Disk Layout of a VM on vSAN
Physical placement of data on vSAN is defined by storage polices, which we will look at in more detail in another section. Here, we look at how the data is placed with vSAN default policy.
Once the VM is created, select the new VM in the inventory, navigate to the Configure tab, and then select Policies. There should be two objects shown, "VM home" and "Hard disk 1". Both should show a compliance status of Compliant meaning that vSAN was able to deploy these objects in accordance with the policy settings.
To verify this, navigate to the Cluster's Monitor tab, and then select Virtual Objects. Once again, both the “VM home” and “Hard disk 1” should be displayed. Select the VM, followed by View Placement Details.
This should display a physical placement of RAID 1 configuration with two components, each component representing a mirrored copy of the virtual disk. It should also be noted that the components are located on different hosts or fault domains. This implies that the policy setting to tolerate 1 failure is being adhered to, as each host is an implicit fault domain. Further, fault domains can be explicitly defined: for instance, hosts within a single rack. Thus, data is resilient to failure of the entire rack. For details on how to create fault domains, see the ‘Configuring Fault Domains’ section.
Physical Placement: vSAN OSA
In a vSAN OSA cluster, the ‘witness’ component is used to maintain a quorum on a per-object basis. For more information, refer to the VMware vSAN Design Guide on core.vmware.com
Physical Placement: vSAN ESA
In vSAN ESA, physical placement is a little different. Data is written into two legs: writes are first ingested into a performance leg and then coalesced and written to a capacity leg. Whilst the distribution of data on the capacity leg reflects the storage policy setting (RAID 1 vs. RAID 5, etc.), the performance leg is always a RAID-1 mirror (and the FTT of the policy is followed). For this reason, RAID-5 performance in vSAN ESA is at least on-par with RAID-1 performance on vSAN OSA. Thus it is recommended, for most workloads, to define a RAID-5 policy for VMs on vSAN ESA.
For more information on object placement in vSAN ESA, visit https://core.vmware.com/vsan-esa
Below we see how the vSAN default policy (RAID-1, FTT-1) distributes the objects (physical disk placement can also be seen per VM, by selecting the VM and navigating to Monitor > vSAN > Physical disk placement). Also note that no witness components are needed (as opposed to vSAN OSA) as there are enough data objects to maintain quorum:
Physical Space Requirements
Note that the “object space reservation” policy setting defines how much space is initially reserved on the vSAN datastore for a VM's objects. By default, it is set to "thin provisioning", implying that the VM’s storage objects are entirely “thin” and consume no unnecessary space. Note the free capacity in the vSAN datastore after deploying the VM, we see that the free capacity is very close to what it was before the VM was deployed, as displayed:
Because we have not installed anything in the VM (such as a guest OS) - it shows that only a tiny portion of the vSAN datastore has so far been used, verifying that the object space reservation setting of "Thin provisioning" is working correctly (observe that the "Virtual disks" and "VM home objects" consume less than 1GB in total, as highlighted in the "Used Capacity Breakdown" section).
Do not delete this VM as we will use it for other tests going forward.
Configuring Fault Domains
As mentioned above, a single host is in itself a fault domain, i.e. data is separated such that the failure of a host does not lead to data loss. On failure, data can be rebuilt elsewhere. We can also group hosts so that data is further spread, and data is better protected.
Fault domains can be defined at cluster creation time (see the sections above). To define (or alter) fault domains thereafter, navigate to [vSAN cluster] > Configure > vSAN > Fault Domains.
Fault domains can then be created by either dragging and dropping into the ‘plus’ icon, or by ticking the appropriate hosts and selecting ‘move hosts’.
Creating a Snapshot
Using the virtual machine created previously, take a snapshot of it. The snapshot can be taken when the VM is powered on or powered off. The objectives are to see that:
- no setup is needed to make vSAN handle snapshots
- the process for creating a VM snapshot is unchanged with vSAN
- a successful snapshot delta object is created
- the policy settings of the delta object are inherited directly from the base disk object
From the VM object in vCenter, click Actions > Snapshots > Take Snapshot...
Take a Snapshot of the virtual machine created in the earlier step.
Provide a name for the snapshot and optional description.
Once the snapshot has been requested, monitor tasks and events to ensure that it has been successfully captured. Once the snapshot creation has completed, additional actions will become available in the snapshot drop-down window. For example, there is a new action to Revert to Latest Snapshot and another action to Manage Snapshots.
Choose the Manage Snapshots option. The following is displayed. It includes details regarding all snapshots in the chain, the ability to delete one or all of them, as well as the ability to revert to a particular snapshot.
To see snapshot delta object information from the UI, navigate to Monitor > vSAN > Physical disk placement.
There are now three objects that are associated with that virtual machine. First is the "VM Home" namespace. "Hard disk 1" is the base virtual disk, and "Hard disk 1 - poc-test-vm1.vmdk" is the snapshot delta. Notice the snapshot delta inherits its policy settings from the base disk that needs to adhere to the vSAN Default Storage Policy.
For vSAN ESA, another object is created (with the same layout as Hard disk 1):
The snapshot can now be deleted from the VM. Monitor the VM’s tasks and ensure that it deletes successfully.
Clone a Virtual Machine
We will continue to use the same VM as before. This time make sure the VM is powered on first.
There are several different cloning operations available with vSAN. Here we will “Clone to Virtual Machine”. The cloning operation is a straightforward click-through operation. This next screen is the only one that requires human interaction. Simply provide the name for the newly cloned VM, and a folder if desired.
We are going to clone the VM in the vSAN cluster, so this must be selected as the compute resource.
On the “Select Storage” screen select the source datastore for the VM, “vsanDatastore”. This will all be pre-selected for you if the VM being cloned also resides on the vsanDatastore.
Select from the available options (leave unchecked - default)
This will take you to the “Ready to complete” screen. If everything is as expected, click FINISH to commence the clone operation. Monitor the VM tasks for the status of the clone operation.
This completes the cloning section of this guide. Do not delete the newly cloned VM, we will be using it in subsequent tests.
vMotion a Virtual Machine Between Hosts
The first step is to power-on the newly cloned virtual machine. We will migrate this VM from one vSAN host to another vSAN host using vMotion.
Note: Take a moment to revisit the network configuration and ensure that the vMotion network is distinct from the vSAN network. If these features share the same network, performance will not be optimal.
First, determine which ESXi host the VM currently resides on. Selecting the Summary tab of the VM shows this, in the ‘related objects window’.
Next, select ‘migrate’, either from the VM ‘actions’ menu or by right-clicking on the VM:
"Migrate" allows you to migrate to a different compute resource (host), a different datastore or both at the same time. In this initial test, we are simply migrating the VM to another host in the cluster, so this initial screen should be left at the default of “Change compute resource only”.
Select Change compute resource only
Then select a destination host:
Select a destination network and click Next.
The vMotion priority can be left as high(default), click Next.
At the “Ready to complete” window, click on FINISH to initiate the migration. If the migration is successful, the summary tab of the virtual machine should show that the VM now resides on a different host.
Verify that the VM has been migrated to a new host:
This completes the “VM migration using vMotion” section of this guide. As you can see, vMotion works just great with vSAN. Do not delete the migrated VM: we will be using it in subsequent tests.
Storage vMotion a VM Between Datastores
This test will only be possible if there is space on a VMFS datastore (such as the boot volume) on the host housing the VM or you have another datastore type available, such as an NFS share. The objective of this test is to successfully migrate a VM from another datastore type into vSAN and vice versa.
Mount an NFS Datastore to the Hosts (optional)
The steps to mount an NFS datastore to multiple ESXi hosts are described in the vSphere Administrators Guide. See the Create NFS Datastore in the vSphere Client topic for detailed steps.
Storage vMotion a VM from vSAN to another Datastore Type
Currently, the VM resides on the vSAN datastore. As we did before, launch the migrate wizard, however, on this occasion move the VM from the vSAN datastore to another datastore type by selecting Change storage only.
Select destination datastore and change the VM Storage Policy to ‘Datastore Default’ as the vSAN policy will not apply to a VMFS (or NFS) datastore. Here we are migrating to the local VMFS datastore on the host:
On the “Ready to complete” screen, click FINISH to initiate the migration.
Once the migration completes, the ‘Datastores’ tab can be used to examine the datastore on which the VM resides.
Verify that the VM has been moved to the new storage.
VM Storage Policies and vSAN
VM Storage Policies form the basis of VMware’s Software-Defined Storage vision. A VM Storage Policy dictates how vSAN should place data (as well as some other features) across the physical resources, such as RAID type, number of stripes and compression (for vSAN ESA). Previously, we have deployed our VMs onto the ‘vSAN Default Storage Policy’.
As the case may be, the actual status of the data may not reflect the policy definition (for instance, if there is a failure, or if the policy has recently been changed). Thus, VM disks can be either compliant or non-compliant with the assigned storage policy. The latter is usually a temporary state, until the system stabilizes. Once the rebuild (or other operation) is complete, compliance is automatically regained.
Note that storage policies are applied per VMDK in vSAN OSA and per VM in vSAN ESA. Further, the recommended storage policy for vSAN ESA clusters is RAID-5 (see the vSAN features guide for more information)
Create a New VM Storage Policy
We will build a policy with RAID 1 and a stripe width of two. The VM Storage Policies can be accessed from the 'Shortcuts' page on the vSphere client, as shown below.
Note: for vSAN ESA, due to the way data is structured, it recommended in most circumstances to define a RAID-5 policy for VMs.
Here we see the existing policies already in place, such as the ‘vSAN Default Storage Policy’ (already used to deploy VMs in the ‘Basic vSphere Functionality’ section of this guide).
To create a new policy, click on Create.
The next step is to provide a name and an optional description for the new VM Storage Policy. Since this policy will contain a stripe width of two, we have given it a name to reflect this. You may also give it a name to reflect that it is a vSAN policy.
The next section sets the policy structure. We select Enable rules for "vSAN" Storage to set a vSAN specific policy
Now we get to the point where we create a set of rules. The first step is to select the availability of the objects associated with this rule. Set the failures to tolerate to one failure (RAID 1)
We then set the Advanced Policy Rules. Once this is selected, the six customizable capabilities associated with vSAN are exposed. Since this VM Storage Policy is going to have a requirement where the stripe width of an object is set to two, this is what we select from the list of rules. It is officially called “Number of disk stripes per object”.
Note: the general recommendation is to keep the number of stripes at the default, one (unless troubleshooting performance or for specific scenarios). Here, we are using this setting to clearly demonstrate how a storage policy affects storage components.
The next screen shows the datastores that are compatible with the policy. In this case, only the vsanDatastore is compatible with the policy settings.
Note: This does not mean that the vSAN datastore can successfully deploy a VM with this policy. It simply means that the vSAN datastore understands the rules or requirements in the policy. Note that the ‘force provisioning’ option will try and apply the policy without first checking if it can be fulfilled by the cluster.
Review the settings on the next screen and click FINISH to create the policy
We can now go ahead and deploy a VM with this new policy and see what effect it has on the layout of the underlying storage objects.
Note: vSAN includes some pre-defined storage polices for the vSAN File Service, named ‘FSVM_Profile_DO_NOT_MODIFY’. These policies are used by internally by vSAN for vSAN file services and should not be modified.
Note: for vSAN ESA, due to the way data is structured, it recommended in most circumstances to define a RAID-5 policy for VMs.
Deploy a new VM with a New Storage Policy
The workflow to deploy a New VM remains the same until we get to the point where the VM Storage Policy is chosen. This time, instead of selecting the default policy, select the newly created policy, as shown below. As before, the vsanDatastore should show up as the compatible datastore, and thus the one to which this VM should be provisioned. To illustrate clearly, we will see how this works on a vSAN OSA cluster to begin with.
Now let's examine the layout of this virtual machine, and see if the policy requirements are met, i.e. do the storage objects of this VM have a stripewidth of 2. First, ensure that the VM is compliant with the policy by navigating to [VM] > Configure > Policies, as shown here.
The next step is to select the [vSAN Cluster] > Monitor > vSAN > Virtual Objects and check the layout of the VM’s storage objects. Select the "Hard Disk 1" object and click on the View Placement Details.
Here, in our vSAN OSA cluster, we can see the two (RAID 0) stripe components, as defined by the policy. Note that each striped component must be placed on its own physical (capacity) disk. There are enough physical disks to meet this requirement here. However, a request for a larger stripe width would not be possible in this configuration. Note that the stripes are across physical disks, and not necessarily disk groups.
However, on examining the “VM Home” object, we see an apparent policy violation – there is no striping seen. This is by design, as there is no performance gain by striping this object, so the policy setting is ignored.
It should also be noted that snapshots taken of this base disk continue to inherit the policy of the base disk, implying that the snapshot delta objects will also be striped.
With a vSAN ESA cluster, the “Hard Disk 1” object is striped, as per the policy on the capacity leg.
Note: for vSAN ESA, due to the way data is structured, it recommended in most circumstances to define a RAID-5 policy for VMs.
And again, the “VM home” object ignores the stripe width from the policy, and the capacity leg is still only RAID 1, single stripe:
Note: the stipe width setting may have unintended consequences if used on an ESA cluster. For more information, visit:
https://core.vmware.com/blog/stripe-width-storage-policy-rule-vsan-esa
Assign a new Storage Policy to an Existing VM
You can choose to modify the VM Storage Policy mapping of an existing VM deployed on the vSAN datastore. The configuration of the objects associated with the VM will be modified to comply with the newer policy. For example, if the number of failures to tolerate (FTT) is increased, newer components would be created, synchronized with the existing object, and subsequently, the original object is discarded. VM Storage policies can also be applied to individual objects.
Here, we will illustrate this (on a vSAN ESA cluster) by applying a new policy to the VM, increasing the failures to tolerate to FTT=2 (keeping RAID 1 and the stripes=2). Once again, we create a new policy. Below, we have named the policy “R1 FTT=2 Stripe Width=2’.
First, we set the FTT value:
Then, as before, the stripe width:
Then, we navigate to our VM, then Configure > Policies and click on EDIT VM STORAGE POLICES
This takes you to the edit screen, where the policy can be changed. The new policy can then be selected from the drop-down list
Once the policy is selected, click the OK button as shown above to ensure the policy gets applied to all storage objects. The VM Storage Policy should now appear updated for all objects. Now when you revisit the Configure > Policies view, you should see the changes in the process of taking effect (Reconfiguring) or completed.
Looking at the physical disk placement, the capacity leg now has three sets of stripes (as expected). Moreover, as we have increased the FTT, the performance leg now has an extra stripe set:
Modify a VM Storage Policy
The workflow above is useful when you only need to modify the policy of one or two VMs, but if you need to change the VM Storage Policy of a significant number of VMs then this can be a little onerous. Instead, we can update by simply changing the policy used by those VMs. All VMs using those policies can then be “brought to compliance” by reconfiguring their storage object layout to make them compliant with the policy. We shall look at this next.
Note: Modifying or applying a new VM Storage Policy leads to additional backend IO as the objects are being synchronized.
In this task, we shall modify an existing VM Storage policy to set the ‘Object Space Reservation’ parameter to 25%. This means that each storage object will now reserve 25% of the VMDK size on the vSAN datastore. Since all VMs were deployed with 40GB VMDKs with Failures to tolerate=1 failure - RAID-1 (Mirroring), the reservation value will be 20 GB.
As the first step, note the amount of free space in the vSAN datastore. This would help ascertain the impact of the change in the policy.
Select StripeWidth=2 policy from the list of available policies, and then the Edit Settings option. Navigate to vSAN > Advanced Policy Rules and modify the Object space reservation setting to 25%, as shown below
Proceed to complete the wizard with default values and click FINISH. A pop-up message requiring user input appears with details of the number of VMs using the policy being modified. This is to ascertain the impact of the policy change. Typically, such changes are recommended to be performed during a maintenance window. You can choose to enforce a policy change immediately or defer it to be changed manually at a later point. Leave it at the default, which is “Manually later”, by clicking Yes as shown below:
Next, select the Storage policy - StripeWidth=2 and click on the VM Compliance tab in the bottom pane. It will display the two VMs along with their storage objects, and the fact that they are no longer compliant with the policy. They are in an “Out of Date” compliance state as the policy has now been changed.
You can now enforce a policy change by navigating to [VM Storage Policies] and clicking on Reapply VM Storage Policy
When this button is clicked, the following popup appears.
When the reconfigure activity completes against the storage objects, and the compliance state is once again checked, everything should show as Compliant.
Since we have now included an ObjectSpaceReservation value in the policy, you may notice corresponding capacity reduction from the vSAN datastore.
For example, the two VMs with the new policy change have 40GB storage objects. Therefore, there is a 25% ObjectSpaceReservation implying 10GB is reserved per VMDK. So that's 10GB per VMDK, 1 VMDK per VM, 2 VMs equals 20 GB reserved space, right? However, since the VMDK is also mirrored, so there is a total of 40GB reserved on the vSAN datastore.
IOPS Limits
vSAN incorporates a quality-of-service feature that can limit the number of IOPS an object may consume. IOPS limits are enabled and applied via a policy setting. The setting can be used to ensure that a particular virtual machine does not consume more than its fair share of resources or negatively impact the performance of the cluster. For more information, visit the following blog: Performance Metrics when using IOPS Limits with vSAN
Here, we demonstrate setting IOPS limits by creating a new policy (as above). We can then set the IOPS limit by navigating to the ‘Advanced Policy Rules’ tab. In our example, we have set the IOPS limit to 1000:
The IOPS limit value is calculated as the number of 32KB IOs per second. Therefore, in this example, where we have a value of 1000, the IOPS limit is 1000x32KB=32MB/s. If I/O against the VM or VMDK should rise above the threshold, the additional I/O will be throttled. Note that any I/O incurred by a snapshot is counted too.
Space Efficiency Defined by Policy (vSAN ESA)
In vSAN ESA compression can be enabled/disabled by VM storage policy. The policy applies to any new data, so turning off compression will only affect any new writes. Note that compression is turned on by default in vSAN ESA.
In the example below, we create a new policy (as per above) and then navigate to the ‘Storage rules’ tab to configure the services. Here, we enable encryption and disable compression:
Note that turning off compression will affect new writes only. Existing data is not affected.
APPENDIX A: RDMA Configuration Example
First, enable PFC support on the switch. The example below shows the configuration steps on a Mellanox SN2100:
Enable PFC:
switch01 [standalone: master] (config) # dcb priority-flow-control enable
This action might cause traffic loss while shutting down a port with priority-flow-control mode on
Type 'yes' to confirm enable pfc globally: yes
Enable Switch PFC for priority 4
dcb priority-flow-control priority 4 enable
Assign PFC to port (ESXi uplink):
switch01 [standalone: master] (config) # interface ethernet 1/9 dcb priority-flow-control mode on force
Verify RDMA available adapter through ESXi shell:
[root@localhost:~] esxcli rdma device list
Name Driver State MTU Speed Paired Uplink Description
------- ---------- ------ ---- -------- ------------- -----------
vmrdma0 nmlx5_rdma Active 4096 100 Gbps vmnic4 MT27700 Family
vmrdma1 nmlx5_rdma Active 4096 100 Gbps vmnic5 MT27700 Family
Looking at each virtual RDMA adapter, we see details on state, MTU size (see hardware specific documentation) and the linked adapter.
Note: To take advantage of RDMA you must have jumbo frames enabled on the physical switch. The RDMA adapter provides <= 4096 (maximum) MTU size.
Verify ESXi RDMA PFC status:
[root@localhost:~]esxcli network nic dcb status get -n vmnic4
Nic Name: vmnic4
Mode: 3 - IEEE Mode
Enabled: true
Capabilities:
Priority Group: true
Priority Flow Control: true
PG Traffic Classes: 8
PFC Traffic Classes: 8
PFC Enabled: true
PFC Configuration: 0 0 0 0 1 1 0 0
If we receive an error here, double check the driver/firmware combination as per vSphere HCL. The vSAN Health check invokes a similar process to query the device DCB status.
Verify ESXi RDMA available protocols:
[root@localhost:~] esxcli rdma device protocol list
Device RoCE v1 RoCE v2 iWARP
------- ------- ------- -----
vmrdma0 true true false
vmrdma1 true true false
Verify vSAN healthcheck in vCenter:
In the screenshot below, we see that PFC is set to the incorrect value, hence the error:
Here we see that PFC is not enabled on the switch:
Verify virtual RDMA adapter performance with esxtop:
SSH to one of the hosts and launch esxtop. Press ‘r’ to show the RDMA screen:
This shows us the throughput for each virtual adapter. Here we see the traffic traversing vmrdma0.
Pressing ‘n’ for the network view shows that there is mimimal traffic on the vmkernel adaptor:
Verify functionality of RDMA and TCP/IP on the same physical vmnic:
In the setup below we can verify that RDMA transport layer is not used for standard TCP/IP protocol and handled separately on the vmnic card layer:
- Enable vSAN RDMA
- Prepare DVS / vSwitch portgroup for VMs using the RDMA adapter
- Configure two VMs with the iperf/iperf3 package installed
- Place both VMs on different hosts
- Run one VM as iperf server (
iperf3 -s
) - Run the second VM as client (
iperf3 -H <IP address of server>)
- On each host, run esxtop and look at the difference between the network (‘n’) and RDMA (‘r’) screens during the iperf3 test
RDMA troubleshooting:
First, verify that the physical network adapters support RDMA. On each host, navigate to Configure > Networking > Physical adapters > [adapter] > RDMA
Note: The RDMA support flag is required for the final setup to enable vSAN RDMA transport (if the RDMA flag is not visible, then double check the hardware specification of the adapter along with driver/firmware versions and the vSphere HCL).
vSphere vSAN RDMA uses RoCEV2 as its protocol layer. When there is no RDMA support available on the physical link or setup, communication falls back to standard legacy TCP/IP automatically.
Esxtop provides additional fields for enablement through the ‘f’ key:
* A: NAME = Name of device
B: DRIVER = driver
C: STATE = State
* D: TEAM-PNIC = Team Uplink Physical NIC Name
* E: PKTTX/s = Packets Tx/s
* F: MbTX/s = Megabits Tx/s
* G: PKTRX/s = Packets Rx/s
* H: MbRX/s = Megabits Rx/s
I: %PKTDTX = % Packets Dropped (Tx)
J: %PKTDRX = % Packets Dropped (Rx)
* K: QP = Number of Queue Pairs Allocated
L: CQ = Number of Completion Queue Pairs Allocated
M: SRQ = Number of Shared Receive Queues Allocated
* N: MR = Memory Regions Allocated
Toggle fields with a-n, any other key to return.
Default setup enables only the minimum requirement for performance for MB/s, queue pairs (QP) and allocated memory regions verbs (MR). For in-depth RDMA functionality, please contact your hardware vendor.
Run the following to obtain detailed adapter statistics. Check for any errors here. Queue pairs are adjusted automatically by requirement:
[root@localhost:~] esxcli rdma device stats get -d vmrdma0
Packets received: 1576258135
Packets sent: 899769661
Bytes received: 40653333761546
Bytes sent: 1079424621290
Error packets received: 0
Error packets sent: 0
Error length packets received: 0
APPENDIX B: FURTHER RESOURCES
Cleanly Removing vSAN Configuration
vCLS Retreat Mode
On occasion, it may become necessary to remove a vSAN cluster and reset hosts to a ‘clean’ state.
To expedite the process, it is advisable to first put vCLS into retreat mode. This will delete the vCLS VMs and make it easier to remove the vSAN datastore and put hosts into maintenance mode, etc.
To achieve this, an vCenter advanced setting, config.vcls.clusters.[domain].enabled
needs to be set.
The procedure to do this is detailed in the documentation here:
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-F98C3C93-875D-4570-852B-37A38878CE0F.html
To make this easier a script is available here to use (download to a Linux or Mac host, uses govc):
https://github.com/vmware-tanzu-experiments/vsphere-with-tanzu-proof-of-concept-samples/blob/main/VCF/vCLS.sh
Remove vSAN Partitions and Clear Data
The next step is to turn off vSAN from vCenter, under [cluster] > Configure > Services > vSAN. If for some reason this step encounters errors, the method below may be useful.
First, open an SSH session to all hosts in the cluster and list the disks used by vSAN by using the command:
vdq -iH
The next step depends on the type of cluster
OSA Clusters
Remove the cache device from each disk group, using the command:
esxcli vsan storage remove -s [cache device]
ESA Clusters
Remove disks from the storage pool, using the command:
esxcli vsan storagepool remove -d [device]
Next, relabel the disks:
partedUtil mklabel /vmfs/devices/disks/[disk] gpt
Again, to make this easier, a script is available to help with this: