vSAN Proof of Concept: vSAN Features

vSAN Space Efficiency Features

Compression Only

vSAN OSA Compression

Beginning with vSphere 7U1, vSAN now supports “Compression Only”. Compression is applied directly at the cluster-level and implemented per disk group. The compression algorithm will take a 4K block and try to compress it to 2KB or less. If this is successful, the compressed block is then written to the capacity tier. If the compression algorithm cannot compress the block by this amount, the full 4KB will be written to the capacity tier.

vSAN ESA Compression

With vSAN ESA, compression is defined at a storage policy level, and is enabled by default. With vSAN ESA, design improvements have increased the compression efficiency, with up to 8:1 compression ratios (per 4K block).

To disable compression, navigate to Home > Policies and Profiles > VM Storage Policies and create or amend an existing policy.

Under the storage rules > space efficiency setting, select ‘no space efficiency’:

Note that this will update new writes only: existing data will not be affected.

Deduplication and Compression (vSAN OSA Only)

In addition to just compressing the data, further savings may be achieved with deduplication in vSAN OSA. When data is destaged from the cache to capacity tier, vSAN will check to see if a match for that block exists. If true, vSAN does not write an additional copy of the block, and metadata is updated. However, if the block does not exist, vSAN will attempt to compress the block.

To demonstrate the effects of Deduplication and Compression, this exercise shows the capacity after deploying a set of identical VMs. Before starting this exercise, ensure that the Deduplication and Compression service is enabled on the cluster. Note that when enabling the Deduplication and Compression service, vSAN will go through a rolling update process: vSAN will evacuate data from each disk group in turn and the disk group will be reconfigured with the features enabled. Depending on the number of disk groups on each host and the amount of data, this can be a lengthy process. 

To enable Deduplication and Compression complete the following steps:

Navigate to [vSAN Cluster] > Configure > vSAN > Services, then select the top EDIT button that corresponds to the Deduplication and compression service:

Toggle Compression only or Deduplication and Compression and select APPLY:  

Testing Compression

Compression rates, and the associated space savings) are very much dependent on the stored data. Further, a data change rates mean that compression ratios are dynamic for a given system. This makes testing compression particularly challenging.

However, we can make reasonable attempts at a repeatable test, given a static, freely available dataset. One such dataset is from the human genome project, hosted by Ensembl (https://www.ensembl.org).

The top-level human genome data consists of a very large text file (comprising of a long string of letters), compressed using gzip. Our test here would be to distribute the data over several VMs on a vSAN datastore and then uncompress the data. Upon enabling compression (and later deduplication and compression for OSA clusters) we can see vSAN compression in action. 

Importing the Dataset

First, create a VM that we can later template. A fast, repeatable method using an Ubuntu image is detailed in Appendix A.

Before the final steps marking the VM as a template and cloning, power on and open an SSH session to the VM. Download the human genome file (around 1GB in size):

curl -u anonymous:password 'https://ftp.ensembl.org/pub/release-108/fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna_sm.toplevel.fa.gz' -o dna.fa.gz

Once this has been downloaded, shutdown the VM and mark as a template. Then clone the VM: the number of clones will, of course, depend on the size of the vSAN datastore.

After cloning, we can inflate the downloaded genome file. The command below uses ‘govc’ utility.
See Appendix A for details on downloading the govc utility.
Note, this command will take some time to execute:

govc find -type m -name 'ubuntu-vm*' | xargs -P0 -I '{}' bash -c 'ssh -o "StrictHostKeyChecking=no" ubuntu@$(govc vm.ip {}) gzip -dv dna.fa.gz'

ESA Cluster

Below is an example on an ESA cluster. Before the decompression phase (i.e. when the VMs have just been cloned), we have around 4.3TB used, which has been compressed to a ratio of around 1.13x (remember that for ESA compression is enabled by default).

After decompressing the dataset on all the VMs, we now have around 11.6TB used, with a compression ratio of around 4x (the consumption here with vSAN compression is greater than with gzip of course. This is balanced with the advantage of accessing and manipulating the data with greater performance):

OSA Cluster – Compression only

Similarly, on an OSA cluster with just compression enabled, we have around 4TB used in the first instance, when the VMs have just been deployed:

After decompression we have around 15.6TB used and a compression rate of around 2x. This is around half of the ratio that we saw from the ESA cluster:

OSA Cluster – Deduplication and Compression

Here we see the effects of enabling deduplication as well as compression. Note that this can take a long time to complete (as each disk group, in turn, will need to be taken offline, formatted and brought online again) to enable the service.

As our dataset is exactly the same across all VMs, we achieve an impressive 19x storage saving across the cluster.

It is important to remember that this is very much an ideal scenario for deduplication. In reality it may be very rare to have such a highly compressible dataset that is exactly homogenous across the cluster.

RAID-5/RAID-6 Erasure Coding

For better space efficiency (compared to RAID-1), we can create storage policies that direct vSAN to use erasure coding. Instead of the 2x or 3x overhead (FTT = 1 or 2) with traditional RAID-1, RAID-5 requires only 33% additional storage, and RAID-6 requires only 50% additional overhead.

In vSAN OSA, to support RAID-5 and RAID-6, the following host requirements must be met:

• RAID-5 (3+1): minimum of four hosts; 1.3x space required
• RAID-6 (4+2): minimum of six hosts; 1.5x space required 

With vSAN ESA, the RAID-5 placement has two separate modes:

• RAID-5 (2+1): three to five hosts; 1.5x space required
• RAID-5 (4+1): minimum of six hosts; 1.25x space required 

Depending on the number of hosts in the cluster, vSAN will automatically adjust the RAID-5 mode. Thus, this ‘adaptive RAID-5’ automatically re-arranges data as the cluster size increases or decreases. We will observe how this works below.

Moreover, the erasure coding architecture in vSAN ESA provides the space savings with the same level of performance as RAID-1. Therefore, for most clusters, the recommended storage policy applied to the VMs should be RAID-5.

For more details on adaptive erasure coding in vSAN ESA, visit:
https://core.vmware.com/blog/adaptive-raid-5-erasure-coding-express-storage-architecture-vsan-8

Below we show how to create a RAID-5 policy and how components are distributed with vSAN OSA and ESA.

Create RAID-5 Storage Policy and Apply

We can easily create a RAID-5 storage policy through vCenter. Navigate to:  Menu > Policies and Profiles > VM Storage Policies:

Next, navigate to 'VM Storage Policies' and 'Create VM Storage Policy':

Select the appropriate vCenter Server and create a name:

 
Ensure 'Enable rules for "vSAN" storage' is checked:

 
Select 'standard cluster' from the 'Site disaster tolerance' drop-down:

 
We see that the vSAN datastore is compatible with this policy:

 
Finally, review and click 'Finish':

Next, we apply this storage policy to an existing VM. Navigate to the VM, select 'Configure' and 'Edit VM storage policies':

Select the newly created storage policy:

 
After this has been set, vSAN will move the data components as per the policy. Once this has been completed, the VM's disks will show as compliant to the policy:

We can now observe the data objects are arranged in vSAN OSA and ESA clusters

RAID-5 Data Placement in vSAN OSA

Navigating to Monitor > Physical disk placement shows the components are now spread over four hosts, i.e. RAID-5 3+1: 

RAID-5 Data Placement in vSAN ESA

As above, we navigate to Monitor > Physical disk placement. As expected, we see the performance leg remain as RAID-1. Moreover, as we have four hosts, vSAN adaptive RAID-5 will select the 2+1 layout for the capacity leg. As shown below, the capacity leg is RAID-5, spread over three hosts, i.e., RAID-5 2+1:

Trim/Unmap

vSAN supports space reclamation on virtual disks using trim commands issued from the guest VM operating system. This functionality is enabled by default in vSAN ESA clusters (and can be explicitly disabled in the VMX file by the setting disk.scsiUnmapAllowed set to false).

Guest Requirements

The following should be met in order for trim/unmap to work:

• At least VM hardware version 11 (Windows) or version 13 (Linux)
• The setting ‘disk.scsiUnmapAllowed’ in the VM’s VMX file set to true (default)
• The VM’s operating system recognizes the disk as ‘thin’
   

For more details, visit:
https://core.vmware.com/resource/vsan-space-efficiency-technologies#sec19560-sub5

Enabling Trim/Unmap on an OSA Cluster

To enable this feature for vSAN OSA, a cluster-wide setting for enabling unmap is set by navigating to [vSAN Cluster] > Configure > Services > Advanced Options > Edit:

Then toggle the ‘Guest Trim/Unmap’ setting. Note that VMs will need to be power cycled for this setting to be effective:

Once unmap is enabled on the cluster, guest VMs can issue commands (such as fstrim) to free any previously deleted data. Trim/unmap is enabled by default on

Example on an OSA Cluster

To demonstrate the effects of this on a vSAN OSA cluster, firstly we observe how much space is in use currently, by navigating to [vSAN Cluster] > [Monitor] > [Capacity].

In this example, we can see that around 931GB of space is currently in use, with around 493GB of VM data

Next, we create or copy a large file on our guest VM. In this case a Windows 2016 VM is used and a large (~76GB) file has been created: 

As expected, our space utilization increases by around 76GBx2 (as this is a RAID-1 object). Thus, 76GBx2 + 493GB gives us around 644GB, as we see below: 

We now delete the file:

Looking back at the capacity view, we can see that the space consumed is still the same:

We now enable the guest trim/unmap advanced setting for the cluster (see above). 

Looking back at vCenter, we see that the used space has been freed:

HCI Mesh (vSAN OSA)

With vSAN OSA, a vSAN datastore can be shared between two OSA clusters, utilizing vSAN’s native data path for cross-cluster connections.

Each vSAN OSA client cluster can mount a maximum of ten remote vSAN OSA datastores. A vSAN OSA server cluster can export its datastore up to a maximum of ten client clusters (see https://configmax.esp.vmware.com for more details).

All vSAN features are supported except for Data-in-Transit encryption, Cloud Native Storage (including vSAN Direct), Stretched Clusters, and 2-Node Clusters. Additionally, HCI Mesh will not support remote provisioning of File Services Shares, iSCSI volumes, or First-Class Disks (FCDs). File Services, FCDs, and the iSCSI service can be provisioned locally on clusters participating in a mesh topology but may not be provisioned on a remote vSAN datastore.

The same MTU sizing is required for both the Client and Server clusters.

New in vSAN 7.0U2, HCI Mesh now supports Compute Only Clusters. Compute only clusters would not require a vSAN license to consume a vSAN datastore on another cluster. Only the cluster providing the storage would need the vSAN license.

To enable HCI Mesh, navigate to [Cluster] > Configure > vSAN > Services and select “I don’t need a local vSAN datastore” 

Next, navigate to Configure > vSAN > Remote Datastores 

This will take you to the remote datastores page (which will show any existing local datastores as well as any remote).

Clicking on ‘mount remote datastore’ will then bring up the following screen. Select the appropriate remote datastore:

On the Check Compatibility screen click Finish. You should now see your new remote datastore available for use.

Encryption in vSAN

There are two (mutually exclusive) modes of encryption available with vSAN, namely data-at-rest and data-in-transit encryption. The former encrypts data on the configured physical devices and the latter across the network. Encryption is enabled and configured at the cluster level.

Encryption is usable alongside all other vSAN features (such as deduplication and compression, RAID-5/6 erasure coding, and stretched cluster configurations among others). In addition, all vSphere features, such as vMotion, DRS, High Availability (HA), and vSphere Replication are supported.

vSAN Data-at-Rest Encryption

Encryption for data at rest became available on vSAN 6.6. Note that self-encrypting drives are not required (check the vSAN Compatibility Guide for device support). 

A Key Management Server (KMS) is required to enable and use data-at-rest encryption. As well as the native (built-in) key provider in vCenter, a third-party solution can be used.

VMware maintains a HCL for KMS servers here: https://www.vmware.com/resources/compatibility/search.php?deviceCategory=kms

These solutions are commonly deployed in clusters of hardware appliances or virtual appliances for redundancy and high availability.

For vSAN OSA, data is encrypted when it is written to persistent media and the encryption step occurs just before the write to the capacity device. In vSAN ESA, the encryption step is higher in the stack, just after the guest write.

Data-at-rest encryption requires that the storage devices are formatted with this option enabled. For vSAN ESA, this can only be done at cluster creation (once a cluster has been created, data-at-rest encryption cannot be enabled).

Conversely, data-at-rest encryption can be enabled in an existing cluster in vSAN OSA. So long as there is enough space in the cluster, data is evacuated from each device, which are then formatted in turn.

Internal Key Management Server

Integrated with vCenter, a built-in ‘native’ key provider is available to use, providing basic key functionality.

To enable, navigate to [Top Level] > Configure > Security Key Providers > Add > Add Native Key Provider

During the naming of the Key Provider, you are presented with the recommended option to only use the key provider with TPM protected ESXi Hosts. Note that it is advisable to use a TPM protected host.

For more details visit:
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-31F2B3D0-259A-4EFD-B675-F89BC27ACE55.html

In order to function, the native key provider must be backed up (a file will be downloaded locally):

When enabling vSAN Encryption, the added Native Key Provider will be available to select. You can add both the internal and external KMS servers to vCenter and use them in different clusters.

Utilizing the Native Key Provider is a quick and easy way to test the vSAN Encryption services. For more advanced functionality, such as redundancy across KMS servers (possibly located in different locations) you may want to consider utilizing external KMS in a cluster configuration.

External Key Management Server

Given the multitude of Key Management Server (KMS) vendors, the setup and configuration of a KMS server/cluster is out of scope of for this document. However, it is a pre-requisite prior to enabling vSAN encryption. The initial configuration of the KMS server is done through vCenter; the KMS cluster is and a trust relationship is established. The process for doing this is vendor-specific, so please consult your KMS vendor documentation prior to adding the KMS cluster to vCenter.

To add an external KMS cluster to vCenter, similar to above, navigate to [Top Level] > Configure > Key Management Servers > Add Standard Key Provider. Enter the information for your specific KMS cluster/server.

Once the KMS cluster/server has been added, you will need to establish trust with the KMS server. Follow the instructions from your KMS vendor as they differ from vendor to vendor.

After the KMS has been properly configured, you will see that the connections status and the certificate have green checks, meaning we are ready to move forward with enabling vSAN encryption.

Enabling vSAN Data-at-Rest Encryption

Prior to enabling vSAN encryption, a KMS must have been deployed (and trusted by vCenter).

To enable vSAN encryption on an existing vSAN OSA cluster, navigate to [vSAN Cluster] > Configure > vSAN > Services, and click EDIT next to the "Encryption" service. Here we have the option to erase all disks before use (Wipe residual data). This will increase the time it will take to do the rolling format of the devices, but it will provide better protection. There is also an option to speed up the process by formatting more than one disk at a time (allow reduced redundancy).

After you click APPLY, vSAN will remove one disk group at a time, format each device, and recreate the disk group once the format has completed. It will then move on to the next disk group until all disk groups are recreated, and all devices formatted and encrypted. During this period, data will be evacuated from the disk groups, so you will see components resyncing.

Note:  This process can take quite some time depending on the amount of data that needs to be migrated during the rolling reformat. If you know encryption at rest is a requirement, go ahead and enable encryption while enabling vSAN. Again, this is the only option if data-at-rest encryption is needed for a vSAN ESA cluster.

Disabling vSAN Data-at-Rest Encryption

Disabling vSAN encryption follows a similar procedure as its enablement. Since the encryption is done at the disk group level, a disk reformat will also be conducted while disabling encryption.

Keep in mind that vSAN will conduct a rolling reformat of the devices by evacuating the disk groups first, deleting the disk group and re-creating the disk group without encryption, at which point it will be ready to host data. The same process is conducted on all remainder disk groups until the vSAN datastore is no longer encrypted.

Since the disk groups are evacuated, all data will be moved within the disk groups, so it may take a considerable amount of time depending on the amount of data present on the vSAN datastore.

Encryption Rekey

You have the capability of generating new keys for encryption. For vSAN OSA, there are two modes for rekeying. The first is a high-level (shallow) rekey where the data encryption key is wrapped by a new key-encryption key. The other level is a complete re-encryption of all data by selecting the option Also re-encrypt all data on the storage using the new keys. This second (deep) rekey may take significant time to complete as all the data will have to be re-written and may decrease performance. For vSAN ESA clusters, only the high-level rekey is possible.

Note:  It is not possible to specify a different KMS server when selecting to generate new keys during a deep rekey; however, this option is available during a shallow rekey.

vSAN Data-in-Transit Encryption

Beginning in vSAN 7U1, we now support Data-in-Transit encryption. Data-in-Transit Encryption can be enabled independently or together with Data-at-Rest encryption to fully protect vSAN data.  Data-in-Transit encryption uses FIPS 140-2 validated VMware VMkernel Cryptographic module.  Both Data and metadata are encrypted.  Unlike Data-at-Rest encryption, Data-in-Transit encryption does not require an external KMS. Keys are managed internally.

Note: if using HCI Mesh mounts or exports, you cannot use Data-in-Transit encryption.

New in vSAN 7.0 Update 1, traffic between data nodes and shared witness can be encrypted by leveraging Data-in-Transit Encryption by simply turning this service on.

When a shared witness node is part of multiple vSAN clusters, Data-in-Transit Encryption feature will be allowed to be enabled only when all nodes in all clusters of shared-witness node have been upgraded to 7.0 U2 version.

Enabling Data-in-Transit Encryption

Enabling Data-in-Transit encryption is easy to do. Select your cluster, then click Configure, and then under vSAN select Services, then click Edit. Tick Data-in-Transit encryption and select your Rekey Interval. The default for rekey interval is one day.  Click Apply.

vSAN File Services (vSAN OSA)

For vSAN OSA clusters, the addition of vSAN File Service quickly enables NFS and SMB shares on vSAN without the need to install or manage a dedicated file service appliance. File shares can be presented to both VMs and containers. Moreover, the entire life cycle of provisioning and managing file services can be seamlessly performed through vCenter.

Cloud Native Use Cases

File services in its first instance is designed to support Cloud-Native workloads. Cloud-Native workloads built on micro-services architecture require data access to be concurrent. Multiple micro-services read and update the same data repository at the same time from different nodes. Updates should be serialized, with no blocking, locking, or exclusivity. This approach differs from the current offering for Cloud-Native storage on vSAN. In the current model, vSAN backed VMDKs are presented to VMs and thus mounted to a single container.

For instance, web services applications like Apache, Nginx, and Tomcat require shared file access to support a distributed operations. Rather than replicating this data to every instance, a single NFS share can be mounted into all containers running these workloads. Hence file storage is critical for Cloud-Native Applications.

vSAN supports extended to SMB v2.1/v3, Kerberos for NFS, and Active Directory

 for SMB and up to 32 hosts per cluster.

The addition of SMB services and Active Directory integration enables file shares in remote locations (e.g. branch offices, etc.) where dedicated Windows file servers are not available. Further use-cases include VDI instances: redirecting user/home directory to a file share location.

In this section we will focus on enabling vSAN File Service, creating and mounting shares, viewing file share properties, and failure scenarios.

Pre-Requisites

Before enabling file services, you will need the following:

• An existing vSAN OSA cluster
• A unique IP address for each file service agent (as per best practice, this will be equal to the number of hosts in the cluster)
• DNS entries (forward and reverse lookup should be working correctly)
• Network details (subnet mask, gateway, etc.)
• Dedicated distributed switch port group (for DVS)
• MAC Learning, MAC change, unknown unicast flooding and forged frames enabled on the segment (for NSX-T)
   

In addition, you will need the following information for the cluster:

• File Services Domain – A unique namespace for the cluster that will be used across shares
• DNS Servers – Multiple DNS entries can be added for redundancy
• DNS Suffix
• Active Directory domain information (for SMB shares or Kerberos authentication)

Below we show an example on how to enable and configure vSAN file services. For full details, visit: 
https://docs.vmware.com/en/VMware-vSphere/8.0/vsan-administration/GUID-CA9CF043-9434-454E-86E7-DCA9AD9B0C09.html

Enabling File Services

vSAN file services are implemented as a set of file server agent VMs (managed by the vSphere ESX Agent Manager). Each agent is a lightweight virtual appliance running Photon OS with a containerized file server. The agent VM is deployed using an OVF file stored in vCenter.

Note: for air-gapped environments, it is advisable to obtain the latest agent OVF file from the link below:
https://customerconnect.vmware.com/en/downloads/info/slug/datacenter_cloud_infrastructure/vmware_vsan/8_0#drivers_tools

For clusters using DVS, It is recommended that a new network port group be created on the distributed switch that the hosts are connected to, as vSAN file services will enable both forged transmits and promiscuous mode. In the example below, we are creating a new port group on VLAN 1002:

Next, navigate to [vSAN OSA Cluster] > Configure > vSAN > Services  the vSAN section. In the list of services, we see that File Services is currently disabled. Begin by clicking Enable.

 

Next, select the port group that was created in the first step:

On clicking enable, the vCenter will deploy the agent VMs. A new resource pool named ‘ESX Agents’ will be created, and the stored OVF deployed and cloned:

Once this process has finished, the service VMs will be visible in the inventory view: 

Note, the file service agent VMs will use a customized storage policy “FSVM_Profile_DO_NOT_MODIFY”.
As the name suggests, do not modify this policy (or assign a different policy to the file service VM).

The next step is to create a file service domain. Navigate again to [vSAN cluster] > Configure > vSAN > Services > File Service and click on CONFIGURE DOMAIN: 

Name the domain. In this example, we have chosen the name ‘vSAN-fs-cluster1’:

On the next screen, enter the networking details (DNS server, suffix, gateway, etc.) and the IP addresses that will be used by the vSAN file service. For consecutive  addresses, use AUTOFILL option to save on typing. Once the IP addresses have been specified, click on LOOKUP DNS to ensure that vCenter can resolve the addresses:

On the next screen, click to enable Active Directory services, if required. Again, this is needed for SMB shares or Kerberos authentication with NFS:

Click review and finally finish on the next screens. If all is well, vCenter will then proceed to enable the vSAN file service.

Creating a File Share

Once file services are enabled navigate to [vSAN Cluster] > Configuration > vSAN > File Shares and click on ADD:

In this example, we are creating an NFS share called ‘app-share’ with the vSAN default storage policy. We have set a 90GB warning for space usage, with a hard quota of 100GB. Additionally, we have created a label ‘mac-open-source’ with the key of ‘applications’:

On the next screen we have a defined specific range of read-only clients (here we have defined a subnet, but a range of IP addresses could also be used):

Click review and finish to create the share. Once vCenter has created to file share, it will be displayed in the File Services Shares:                           

After the share has been created any quota, labels or network permissions can be made by selecting the file share and selecting EDIT.

Mounting a File Share

First, obtain the path of the share by navigating to [vSAN Cluster] > Configure > vSAN > File Shares. Select the appropriate file share and click on COPY PATH:

We can then mount the share as desired, for instance:

$ mount db-fs1.vsanpe.vmware.com:/vsanfs/app-share /mnt/app-share 

Quotas and Health Events

During file share creation, quotas and hard limits can be set. In this sample file share, a warning threshold of 90 GB was specified and a hard limit of 100GB was set. As part of this test copy some data to the file share to fill the space required to trigger the quota. Once the warning threshold is exceeded the Usage over the Quota field in the UI will turn red.

Once the hard quota is reached writes to the share will fail with a disk quota exceeded error as shown below. 

cp: error writing 'file13.txt': Disk quota exceeded 

If the quota is reached an alarm in Skyline Health is also triggered. The details of the alarm can be viewed by expanding the File Services section.

Once data has been removed from the Share Health alert is cleared and the File Service health reports as normal.

Failure Scenarios

Storage policies apply to file service objects just as they do other virtual disk objects. Health and placement details of file shares are shown in the Virtual Objects view.

By clicking VIEW PLACEMENT DETAILS the layout of the underlying vSAN object can be viewed. This view shows component status, and on which hosts components of the share reside.

To test host failure, we will power off one of the hosts containing an active copy of the file share data. Once the host is powered off, we see that the component of the corresponding host is displayed as absent.                          
 

Now that the host has been shut down, you can validate from any of the client virtual machines through a file browser or logs to verify that file share is still accessible.

File Services Snapshots

From vSAN 7 Update 2, vSAN includes a new snapshotting mechanism allowing for point-in-time recovery of files. Snapshots for file shares can be created through the UI. Recovery of files is available through API allowing backup partners to extend current backup platforms to protect vSAN file shares.

File Services Support for Stretched Clusters and Two Node Topologies

File services can now be used in vSAN stretched clusters and two node topologies. The site affinity setting for file hares defines where the presentation layer resides. Site affinity for file shares is defined by the storage policy associated with the individual file shares. The storage policy and site affinity settings to be applied to the file share are defined as part of the creation process.

The image below is an example of the site affinity setting available when creating a file share in a stretched cluster.

Cloud-Native Storage for Vanilla K8s

Note: this section details the Cloud Native Storage (CNS) features  in vSAN for standard or ‘vanilla’ Kubernetes clusters. See the link below for more information on CNS for vSphere:
https://blogs.vmware.com/virtualblocks/2019/08/14/introducing-cloud-native-storage-for-vsphere/

Conversely, Tanzu Kubernetes Grid (TKG) is a Kubernetes runtime created by VMware. For more information on using storage features VMware Tanzu Kubernetes, visit:

https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-concepts-planning/GUID-E297DD43-AEEB-4B6D-8C93-4212CA62309A.html#GUID-E297DD43-AEEB-4B6D-8C93-4212CA62309A

Overview

Cloud-Native Storage (CNS), introduced in vSphere 6.7 Update 3, offers a platform for stateful cloud-native applications to persist state on vSphere backed storage. The platform allows users to deploy and manage containerized applications using cloud-native constructs such as Kubernetes persistent volume claims and maps these to native vSphere constructs such as storage policies. CNS integrates with vSphere workflows and offers the ability for administrators to perform tasks such as defining storage policies that could be mapped to storage classes, list/search and monitor health and capacity for persistent volumes (PV). vSphere 6.7 Update 3 supports PVs backed by block volumes on vSAN as well as VMFS and NFS datastores. However, some of the monitoring and policy-based management support may be limited to vSAN deployments only. With the release of vSAN 7.0, PVs can also be backed by the new vSAN File Service (NFS). 

Cloud Native Storage Prerequisites

While Cloud Native Storage is a vSphere built-in feature that is enabled out-of-the-box starting in vSphere 6.7 Update 3, it is required to install the Container Storage Interface (CSI) driver in Kubernetes to take advantage of the CNS feature. CSI is a standardized API for container orchestrators to manage storage plugins. CNS support requires minimum version of Kubernetes v1.14 release. Configuration procedure of CSI driver in Kubernetes is beyond the scope of this guide. To learn more about the installation of the CSI driver, refer to the vSphere CSI driver documentation.

Deploy a Persistent Volume for Container (Block Type)

Assuming your Kubernetes cluster has been deployed in the vSAN cluster and the CSI driver has been installed, you are ready to check out the Cloud Native Storage functionality.

First, we need to create a yaml file like below to define a storage class in Kubernetes. Notice the storage class name is “cns-test-sc” and it is associated with the vSAN Default Storage Policy.  Note also that the “provisioner” attribute specifies that the storage objects are to be created using the CSI driver for vSphere block service.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cns-test-sc
  provisioner: block.vsphere.csi.vmware.com

parameters:
  StoragePolicyName: "vSAN Default Storage Policy"

kubectl apply -f cns-test-sc.yaml

kubectl get storageclass cns-test-sc
NAME          PROVISIONER                    AGE
cns-test-sc   block.vsphere.csi.vmware.com   20s

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cns-test-pvc
  labels:
    app: cns-test
    release: cns-test
spec:
  storageClassName: cns-test-sc
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

kubectl apply -f cns-test-pvc.yaml

kubectl get pvc cns-test-pvc
NAME           STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cns-test-pvc   Bound    pvc-ebc2e95c-b98f   2Gi        RWO            cns-test-sc    15s

Deploy a Persistent Volume for Container (File Type)

Assuming your Kubernetes cluster has been deployed in the vSAN cluster and the CSI driver (version 2.0 or above is required to support ReadWriteMany access mode) has been installed, and vSAN file service is enabled, you are ready to check out the Cloud Native Storage functionality for File.

First, we need to create a yaml file like below to define a storage class in Kubernetes. Notice the storage class name is “vsan-file” and it is associated with the vSAN Default Storage Policy.  Note also that the “fstype” attribute specifies that the storage objects are to be created using the CSI driver for vSAN file service.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: vsan-file
  provisioner: csi.vsphere.vmware.com
parameters:
  storagePolicyName: "vSAN Default Storage Policy"
  csi.storage.k8s.io/fstype: nfs4

Apply the storage class by executing the following command on your Kubernetes master node:

kubectl apply -f file-sc.yaml

Run the command below to confirm the storage class has been created.

kubectl get storageclass vsan-file
NAME        PROVISIONER              AGE
vsan-file   csi.vsphere.vmware.com   20s

Next, we need to create another yaml file like below to define a Persistent Volume Claim (PVC). For illustration purposes we simply create a persistent volume without attaching it to an application. Notice “storageClassName” references to the storage class that was created earlier. There are two labels assigned to this PVC: app and release. We will see later how they get propagated to CNS in the vSphere Client UI.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: file-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
  storageClassName: vsan-file

Create the PVC by executing the following command on your Kubernetes master node:

kubectl apply -f file-pvc.yaml

Run the command below to confirm the PVC has been created, its status is listed as “Bound”, and access mode is set for "RWX" to allow multiple pods to read from and write to the PV simultaneously.

kubectl get pvc file-pvc

To examine what container volume has been created in vSphere Client UI, navigate to [vSAN cluster] > Monitor > Cloud Native Storage > Container Volumes. You should see a container volume with a name that matches the output from the “get pvc” command. 624x136​

More information such as Kubernetes cluster name, PVC name, namespace, and other storage properties from vSphere perspective is available by clicking on the text icon next to the name:

624x300​

If you want to investigate the overall usage of all container volumes in the cluster, navigate to [vSAN cluster] > Monitor > vSAN > Capacity. The Capacity view breaks down the storage usage at a granular level of container volumes that are either attached or not attached to a VM.

624x298​ 

APPENDIX A: Creating Test VMs

Here we demonstrate how to quickly create a set of identical VMs for testing.

For a more verbose procedure, visit Myles Gray’s blog: 
https://blah.cloud/infrastructure/using-cloud-init-for-vm-templating-on-vsphere

Requirements:

• FreeBSD, Linux or MacOS VM/host environment
• Latest version of govc (download instructions below)

Download govc:

Govc is a lightweight, open-source CLI tool written in Go (and part of the Govmomi/Go library for the vSphere API). Project page: https://github.com/vmware/govmomi/tree/master/govc

To download the latest release, use the command below, or visit the release page: https://github.com/vmware/govmomi/releases

As with the majority of Go projects, it is packaged as a single binary (note that the tar command requires root privileges to copy the binary to the correct location):

curl -L -o - "https://github.com/vmware/govmomi/releases/latest/download/govc_$(uname -s)_$(uname -m).tar.gz" | tar -C /usr/local/bin -xvzf - govc

Connecting to vCenter

To authenticate with vCenter, we need to define the username, password and URL, as per the example below:

export GOVC_USERNAME=administrator@vsphere.local
export GOVC_PASSWORD=P@ssw0rd
export GOVC_INSECURE=1
export GOVC_URL=10.156.163.1

Additionally, we will need to specify the default datastore and resource pool (we can define this as the default/top-level cluster, as per blow) for deploying our VMs:

export GOVC_DATASTORE=ESA-vsanDatastore
export GOVC_RESOURCE_POOL='vSAN ESA Cluster/Resources'

Finally test the connection to vCenter by issuing the command below, it should return with details:

govc about
FullName:     VMware vCenter Server 8.0.0 build-20519528
Name:         VMware vCenter Server
Vendor:       VMware, Inc.
Version:      8.0.0
Build:        20519528
...

Configure Test VM

First, specify a location of an OVA file to use. In the example below, we use an Ubuntu 22.04 cloud image:

export vmLocation=https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.ova

We can then add our customizations, etc. by extracting the JSON from the OVA:

govc import.spec $vmLocation > ubuntu-vm.json

Ubuntu uses cloud-init to setup the OS environment. As we will be cloning the deployed VM, we need to define specific user-data (which will be encoded in base-64 and added to the customization JSON). Here we ensure that vSphere specific configuration is not disabled, and we modify the default netplan configuration file to ensure DHCP addresses are assigned by mac address (rather than machine-id).

To simplify the process, the user-data file can be downloaded from the link below:
https://raw.githubusercontent.com/vmware-tanzu-experiments/vsphere-with-tanzu-proof-of-concept-samples/main/VCF/test_vms/user-data

#cloud-config
runcmd:
  - 'echo "disable_vmware_customization: false" >> /etc/cloud/cloud.cfg'
  - echo -n > /etc/machine-id
  - |
    sed -i '' -e 's/match.*/dhcp-identifier: mac/g' -e '/mac/q' /etc/netplan/50-cloud-init.yaml
final_message: "The system is prepped, after $UPTIME seconds"
power_state:
  timeout: 30
  mode: poweroff

If available, use the cloud-init CLI to check the user-data file:

base64 -i user-data

Now we can edit the JSON file we extracted earlier. Change the file with the following:

• Disk provisioning set to ‘thin’
• Add the public key of the machine we are connecting from
• Remove the hostname and password data
• Set the network for the VM (the name of the relevant portgroup in vCenter)
• Set the name of the VM
• In the ‘user-data’ section, paste in the  base64 encoded data

Note we can avoid hand-editing the JSON by using jq. For example, we can create a new JSON with user-data:

jq --arg udata "$(base64 -i user-data)"  '.PropertyMapping[4].Value=$udata' ubuntu-vm.json > u1.json

Similarly, adding a public key stored in a user's GitHub profile (replace [GITHUB-USERID] with your GitHub username):

jq --arg user "$(curl -sk https://api.github.com/users/[GITHUB-USERID]/keys | jq -r '.[].key')"  '.PropertyMapping[3].Value=$user' u1.json > updated_ubuntu.json

An example of this file can be seen here:
https://raw.githubusercontent.com/vmware-tanzu-experiments/vsphere-with-tanzu-proof-of-concept-samples/main/VCF/test_vms/ubuntu-vm.json
 

{
  "DiskProvisioning": "thin",
  "IPAllocationPolicy": "dhcpPolicy",
  "IPProtocol": "IPv4",
  "PropertyMapping": [
    {
      "Key": "instance-id",
      "Value": "id-ovf"
    },
    {
      "Key": "hostname",
      "Value": ""
    },
    {
      "Key": "seedfrom",
      "Value": ""
    },
    {
      "Key": "public-keys",
      "Value": "ssh-rsa AAAAB3NzaC1yc2EAAAAD..."
    },
    {
      "Key": "user-data",
      "Value": "I2Nsb3VkLWNvbmZpZwpy..."
    },
    {
      "Key": "password",
      "Value": ""
    }
  ],
  "NetworkMapping": [
    {
      "Name": "VM Network",
      "Network": "DSwitch-DHCP"
    }
  ],
  "MarkAsTemplate": false,
  "PowerOn": false,
  "InjectOvfEnv": false,
  "WaitForIP": false,
  "Name": "ubuntu-vm"
}

Once this JSON file has been defined, we can double-check our user-data encoding is still correct:

awk -F '"' '/user-data/{getline; print $4}' ubuntu-vm.json | base64 -d

This should return the user-data as we defined above.

Import OVA to vCenter and Clone

We can then import the OVA into vCenter, specifying our JSON customization file:

govc import.ova -options=ubuntu-vm.json -name=ubuntu-template $vmLocation

After this has imported, we can update the virtual disk size. Here we set it to 100G:

govc vm.disk.change -vm ubuntu-template -disk.label "Hard disk 1" -size 100G

Power on the VM to allow it to run cloud-init (and thus our previously defined commands). Once complete, the VM will shutdown:

govc vm.power -on ubuntu-template

Once the VM has shutdown, mark it as a template:

govc vm.markastemplate ubuntu-template

Finally, we can clone our template VM as we need to. In the example below, we clone it ten times:

for x in {1..10};do govc vm.clone -vm ubuntu-vm ubuntu-vm$x;done

To do this for a large number of VMs, in parallel (and output to a log file) we could run:

for x in {1..250};do (govc vm.clone -vm ubuntu-template ubuntu-vm$x >> $(date +%d%m-%H%M)_clone.log 2>&1 &);done

We can monitor progress by probing the vCenter task-list:

govc tasks -f -l

After cloning, we can batch-execute commands on all the VMs. For example, the 'ls' command:

govc find -type m -name 'ubuntu-vm*' | xargs -P0 -I '{}' bash -c 'ssh -o "StrictHostKeyChecking=no" ubuntu@$(govc vm.ip {}) ls'