]

Solution

  • Storage

Type

  • Document

Level

  • Intermediate

Category

  • What's New

Product

  • Site Recovery
  • Site Recovery Manager
  • Site Recovery Manager 8
  • vSphere
  • vSphere 7

Technology

  • Cloud Native Storage
  • Kubernetes
  • Raw Device Mapping (RDM)
  • Virtual Volumes (vVols)
  • VMFS

What's New in vSphere 7 Core Storage

PSA

NVMeoF

Insight

 NVMe continues to become more and more popular because of its low latency and high throughput. Industries, such as Artificial Intelligence, Machine Learning, and IT, continue to advance, and the need for increased performance continues to grow. Typically, NVMe devices are local using the PCIe bus. So how can you take advantage of NVMe devices in an external array? The industry has been advancing external connectivity options using NVMe over Fabrics (NVMeoF). Connectivity can be either IP or FC based. There are some requirements for external connectivity to maintain the performance benefits of NVMe as typical connectivity is not fast enough.

 

VMware NVMeoF

 In vSphere 7, VMware added support for shared NVMe storage using NVMeoF. For external connectivity, NVMe over Fibre Channel and NVMe over RDMA (RoCE v2) are supported.

 

With NVMeoF, targets are presented as namespaces, which is equivalent to SCSI LUNs, to a host in Active/Active or Asymmetrical Access modes. This enables ESXi hosts to discover, and use the presented NVMe namespaces. ESXi emulates NVMeoF targets as SCSI targets internally and presents them as active/active SCSI targets or implicit SCSI ALUA targets.

 

NVMe over Fibre Channel

 

This technology maps NVMe onto the FC protocol enabling the transfer of data and commands between a host computer and a target storage device. This transport requires an FC infrastructure that supports NVMe.

 To enable and access NVMe over FC storage, install an FC adapter supporting NVMe in your ESXi host. There is no configuration required for the adapter; it will automatically connect to an appropriate NVMe subsystem and discovers all shared NVMe storage devices. You may, at a later time, reconfigure the adapter and disconnect its controllers or connect other controllers.

NVMe over FC Requirements

  • NVMe array supporting FC
  • Compatible vSphere 7 ESXi host
  • HW NVMe adapter (HBA supporting NVMe)
  • NVMe controller

 

NVMeoF over RDMA

 This technology uses Remote Direct Memory Access (RDMA) transport between two systems on the network. The transport enables in memory data exchange, bypassing the operating system or processor of either system. ESXi supports RDMA over Converged Ethernet v2 (RoCE v2).

 To enable and access NVMe storage using RDMA, the ESXi host uses an RNIC adapter on your host and a SW NVMe over RDMA storage adapter. You must configure both adapters to use them for NVMe storage discovery.

 

NVMe over RDMA requirements:

 

NVMeoF Setup Prerequisites

When setting up NVMeoF, there are a few practices that should be followed.

  • Do not mix transport types to access the same namespace.
  • Ensure all active paths are presented to the host.
  • NMP is not used/support; instead, HPP (High-Performance Plugin) is used for NVMe targets.
  • You must have dedicated links, VMkernels, and RDMA adapters to your NVMe targets.
  • Dedicated layer 3 VLAN or layer 2 connectivity
  • Limits:
    • Namespaces-32
    • Paths=128 (max 4 paths/namespace on a host)

Shared VMDK

 In vSphere 7, VMware added support for SCSI-3 Persistent Reservations (SCSI-3 PR) at the virtual disk (VMDK) level. What does this mean? You now have the ability to deploy a Windows Server Failover Cluster (WSFC), using shared disks, on VMFS. This is yet another move to reduce the requirement of RDMs for clustered systems. With supported HW, you may now enable support for clustered virtual disks (VMDK) on a specific datastore. Allowing you to migrate off your RDMs to VMFS and regain much of the virtualization benefits lost with RDMs.

Shared VMDKs on VMFS Prerequisites

  • Your array must support ATS, SCSI-3 PR type Write Exclusive-All Registrant (WEAR).
  • Only supported with arrays using Fibre Channel (FC) for connectivity.
  • Only VMFS6 datastores.
  • Storage devices must be claimed by NMP. ESXi does not support third-party plug-ins (MPPs) in clustered virtual disk configurations.
  • VMDKs must be Eager Zeroed Thick (EZT) Provisioned.
  • Clustered VMDKs must be attached to a virtual SCSI controller with bus sharing set to “physical.”
  • A DRS anti-affinity rule is required to ensure VMs, nodes of a WSFC, run on separate hosts.
  • Change/increase the WSFC Parameter "QuorumArbitrationTimeMax" to 60.

 

Other Caveats

  • Windows Server 2012R2/2016/2019. SQL Server 2016/2017 was used to validate the configuration.
  • The boot disk (and all non-shared disks) should be attached to a separate virtual SCSI controller with bus sharing set to “none.”
  • Mixing clustered and non-shared disks on a single virtual SCSI controller is not supported.
  • The datastore cannot be expanded or span multiple extents.
  • All hosts and vCenter must be vSphere 7 or above
  • A mix of clustered VMDKs and other disk types (vVols, RDMs) are not supported
  • Limits:
    • Support for up to 5 WSFC nodes (Same as RDMs)
    • 128 clustered VMDKs per host
  • Only Cluster across Box (CaB) is supported, Cluster in a Box (CiB) is not supported.

 

vSphere Features

  • Supported Features:
    • vMotion to supported hosts meeting the same requirements.
  • Unsupported Features:
    • Snapshots, cloning, and Storage vMotion
    •  Fault Tolerance (FT)
    • Hot change to VM HW or hot expansion of clustered disks

 

Here’s a link to VMware’s document on Microsoft Clusters.

Enabling Clustered VMDK

 When you navigate to your supported datastore, under the Configure tab, you will see a new option to enable  Clustered VMDK. If you are going to migrate or deploy a Microsoft WSFC cluster using shared disks, then you would enable this feature. Once the feature is enabled, you can then follow the Setup for Windows Server Failover Clustering documentation to deploy your WSFC on the VMFS6 datastore.

Enabling Clustered VMDK

When you navigate to your supported datastore, under the Configure tab, you will see a new option to enable  Clustered VMDK. If you are going to migrate or deploy a Microsoft WSFC cluster using shared disks, then you would enable this feature. Once the feature is enabled, you can then follow the Setup for Windows Server Failover Clustering documentation to deploy your WSFC on the VMFS6 datastore.

 

 

Perennially Reserved Flag for RDMs

Using the Perennially Reserved Flag for WSFC RDMs

In cases where customers are using numerous pRDMs in their environment, host boot times or storage rescans can take a long time. The reason for the longer scan times is each LUN attached to a host is scanned at boot or during a storage rescan. Typically, RDMs are provisioned to VMs for Microsoft WSFC and are not directly used by the host. During the scan, ESXi attempts to read the partitions on all the disks but it is unable to for devices persistently reserved by the WSFC. Subsequently, the longer it can take a host to boot or rescan storage. The WSFC uses SCSI-3 persistent reservation to control locking between the nodes of the WSFC which, blocks the hosts from being able to read them.

VMware recommends implementing perennial reservations for all ESXi hosts hosting VM nodes with pRDMs. Check the following kb1016106 or more details

The question then arises; How can you get the host not to scan these RDMs and reduce boot or rescan times? I’m glad you asked!

There is a device flag called “Perennially Reserved” which tells the host the RDM should not be scanned and is used elsewhere (perennially) in the environment. Before vSphere 7, this flag is enabled via CLI and requires the UUID (naa.ID).

The command to set the flag to true:
esxcli storage core device setconfig -d naa.id --perennially-reserved=true

To verify the setting:
esxcli storage core device list -d naa.id

In the device list you should see:
Is Perennially Reserved: true

When setting this option, it must be run for each relevant RDM used by the WSFC and on every host with access to that RDM. You can also set the Perennially Reserved flag in Host Profiles.

With the release of vSphere 7, setting the Perennially Reserved flag to true was added to the UI under storage devices. There has also been a field added to show the current setting for the Perennially Reserved flag.

Once you select the RDM to be Perennially Reserved, you have the option to “Mark the selected device as perennially reserved” for the current host or multiple hosts in the cluster. This eliminates the manual process of setting the option per host via CLI. If preferred, you can still use ESXCLI, PowerCLI, or Host Profiles.

 

Once you click YES, the flag will be set to true and the UI updated.

 

You may also unmark the device using the same process.

 

Short Clip showing process:

 

 

Setting the Perennially Reserved flag on your pRDMs used by your WSFC is recommend in the clustering guides. When set, ESXi no longer tries to scan the devices and this can reduce boot and storage rescan times. I have added links below to resources on clustering guides and the use of this flag. Another benefit of flagging RDMs is you can easily see which devices are RDMs and which are not.

@jbmassae

Resources:

  • ESXi/ESX hosts with visibility to RDM LUNs being used by MSCS/WSFC nodes with RDMs may take a long time to start or during LUN rescan (1016106)
  • Guidelines for Microsoft Clustering on vSphere (1037959)
  • Microsoft Windows Server Failover Clustering (WSFC) with shared disks on VMware vSphere 7.x: Guidelines for supported configurations (79616)
  • Microsoft Windows Server Failover Clustering (WSFC) with shared disks on VMware vSphere 6.x: Guidelines for supported configurations (2147661)

VMFS

Affinity 2.0

Overview

 Some of the benefits of deploying VMs as Thin Provisioned VMDKs are the effective use of space, and space reclamation. A thin VMDK is a file on VMFS, where Small File Blocks (SFB) are allocated on-demand at the time of first write IO. There can be an overhead cost to this process, which can affect performance. In some cases, for maximum performance, it is recommended Eager Zero Thick (EZT) disks are utilized to avoid the overhead of allocating space for new data.

First Write Process

 In VMFS, resources are organized, on-disk, in groups called “Resource Clusters” (RC). When a file requires new storage resources, (new data to be written), there are two decisions made.

1. Which RC to fetch the resources.

2. Which resources from the RC to allocate.

The “Affinity Manager” is a component of VMFS6 responsible for RC allocation. The Affinity Manager allocates RCs, particularly Small File Block Cluster (SFBC), such that minimal files share the same RC. The process of mapping a resource to an RC is called affinitizing. VMFS is shared storage, and multiple hosts can request resources in parallel. Resource allocations are synchronized using an on-disk lock (ATS) with no other communication between hosts. This can lead to issues during allocation. The Resource Manager requests RCs from the Affinity manager, who then responds to the VMFS file layer.

New Affinity 2.0 Manager

 With vSphere 7, we are introducing Affinity 2.0, which is designed to be smarter and more efficient in the allocation of resources, minimizing overhead in the IO path. This  is accomplished by creating a cluster-wide view of the resource allocation status of the RCs, dividing disk space into “regions.” Then ranking them based on the number of blocks allocated by the current host as well as other hosts. This information is called a “Region Map.” With the Region Map, the Affinity Manager can now quickly supply information on available RCs to the Resource Manager.

Benefit

 What does all this mean? When the Resource Manager requests RCs from the new Affinity Manager, when a first write IO occurs, the allocations are no longer a “go and find” an RC to allocate avoiding the back and forth overhead. With the Region Map, the Affinity Manager knows where and what RC resources are available and can quickly direct the Resource Manager. The Resource request, from the VMFS file layer, now goes directly to the Affinity Manager. The Affinity Manager relies on the generated Region Map to find an available RC to allocate. The Affinity Manager then locks the RC on-disk, checks for free resources, and hands it over to the Resource Manager. This reduces the repeated back and forth between the Affinity Manager and Resource Managers trying to find an available RC. Therefore, reducing metadata IO and the overhead required for the first write on thin-provisioned disks. It can also improve the allocation of space for EZT or LZT provisioning.

 

End Of Availability (EOA)

With the release of vSphere 7, there are a few antiquated features that have reached End of Availability.

vFRC

vFRC currently has a minimal customer base. With VAIO, it allows 3rd party vendors to create custom caching solutions. When you upgrade to vSphere 7, you will receive a warning message that vFRC will no longer be available.  "vFRC will be gone with this upgrade, please disable vFRC on a VM if using it."

 

CBRC 1.0 EOA

CBRC 1.0 has a maximum cache size of 2GB, whereas 2.0 has a maximum of 32GB. As of vSphere 6.5, CBRC 2.0 is the default for Content-Based Read Cache. Starting in vSphere 7, CBRC 1.0 has been removed to ensure it is not used, especially in Horizon environments. This will also eliminate the building and compiling of un-used code.

 

vVols Interoperability

 VMware Virtual Volumes (vVols) adoption continues to grow and is accelerating in 2020, and it’s easy to see why. vVols eliminates LUN management, accelerates deployments, simplifies operations, and enables utilization of all of your array’s functionality. VMware and our storage partners continue to develop and advance vVols, and it’s functionality. In vSphere 7, more features and enhancements have been added, showing the continued commitment to the program.

vVols Support in SRM 8.3

SRM

Because vVols uses array-based replication, it is very efficient. Array-based replication is a preferred method of replicating data between arrays. With vVols and SPBM, you can easily manage which VMs are replicated rather than everything in a volume or LUN. With the release of Site Recovery Manager 8.3, you can now manage your DR process with SRM while using the replication efficiency and granularity of vVols and SPBM.

Here’s a link to the announcement blog for SRM 8.3

 

vVols Support for CNS

CNS

 Kubernetes continues to grow in adoption, and VMware is on the forefront. One of K8s’ requirements is persistent storage, and until now, that included vSAN, NFS, and VMFS. The thing is vVols couldn’t be more suited for K8s storage because a vVol is its own entity. Now, deploy an FCD as a vVol and you’ve got a first-class disk as a first-class citizen that has additional benefits like mobility and CSI to SPBM policy mapping. With the initial release, snapshots and replication with vVols will not be supported.

 

vVols Support in vRealize Operations 8.1

vRealize Operations

A feature that has been requested for a while is finally available; support for vVols datastores in vROps! With the release of vROps 8.1, you can now utilize vROps monitoring on your vVols datastores the same as any other datastore. Giving your alerting, planning, troubleshooting, and more for your vVols datastores. For more information here's the link to vROps.
Make sure to read about the new release on the vROps 8.1 announcement blog.

 

vVols as Supplemental Storage in VCF

VCF

VMware Cloud Foundation allows organizations to deploy and manage their private and public clouds. VCF currently supports vSAN, VMFS, and NFS principle storage. Customers are asking for support of vVols as principle storage, and while the VCF team continues to evaluate and develop that option, it is not available. In the meantime, vVols can be used as supplemental storage after the Workload Domain build has completed. Support for vVols as supplemental storage is a partner supported option.

Please work with your storage array vendor for the supported processes and procedures in setting up vVols with VCF as supplemental storage.

For more information, here’s the link to VCF.

Here’s a link to the blog on What’s New in VCF 4.0

 

Filter Tags

  • Storage
  • Intermediate
  • What's New
  • Document
  • Site Recovery
  • Site Recovery Manager
  • Site Recovery Manager 8
  • vSphere
  • vSphere 7
  • Cloud Native Storage
  • Kubernetes
  • Raw Device Mapping (RDM)
  • Virtual Volumes (vVols)
  • VMFS