vSAN File Services

Introduction

File services is a common requirement in today’s enterprise environments. The need for file-level access can come from a variety of use cases - from traditional systems needing to store user directories, to modern cloud-native applications demanding persistent, read-write-many (RWM) volumes.

Historically, providing these file-level services meant using physical storage arrays or VMs capable of serving file-level protocols such as NFS and SMB. These approaches introduce additional factors in the design and management of an environment that are not trivial.

vSAN native file services draws attention for many reasons. Its flexibility, integration, and capabilities make it a good fit for a variety of use cases. The initial version provided a key element to serving up cloud-native applications in vSAN: Persistent, read-write many (RWM) volumes. vSAN 7 Update 1 improved on the capabilities of file services even further with support for SMB v2.1 and v3, Active Directory integration and Kerberos Support. vSAN 7 U2 extends the capabilities of vSAN file services in new and interesting ways including support for stretched clusters, data-in-transit encryption, snapshots and improved scale, performance and efficiency.

Use Cases

With vSAN 7 Update 2, there are several key use cases that were targeted for file services including, home directories, user profiles, infrastructure share needs, cloud-native workloads, physical and virtual workloads needing file services.

For more information on use cases, limitations and capabilities please see the vSAN File Services FAQ.

Architecture

File services is a cluster-level service and can be enabled in the same location of the UI as the other cluster-level services such as the iSCSI service, deduplication and compression, and data-at-rest encryption. File services are rendered through a series of containers deployed in the cluster. The containers run inside photon powered VMs deployed across the cluster. These container hosts are deployed from a lightweight OVF that is cached on the vCenter server after download. They are powered off and deleted if file services are disabled on the cluster but will run on every host in the cluster when the service is enabled.

Architecture overview

Let’s look at the basic architecture of vSAN file services. When the vSAN file services feature is enabled, this will initiate the deployment of stateless containers that provide the file system protocol desired – in this case, NFSv3, NFSv4, SMB2, and SMB3. The protocol services containers are spread across the cluster to provide the resilience of the service and is not something that the user needs to manage. Up to 32 hosts per cluster contribute to the processing of file services

Stretched Cluster support for vSAN File Shares

The site affinity setting for a file share is defining where the presentation layer (NFS or SMB services) reside. It does not relate to if (or how) the data is placed across sites. This is defined by the storage policy associated with the share. The storage policy settings protect the share data in the same manner as any other type of object data, such as no site-level mirroring, site-level mirroring, and site-level mirroring with secondary levels of protection at each site including RAID-1 and RAID-5/6.

Scaling SMB and NFS Shares

Expanding front end access

The VMware vSAN Virtual Distributed File System (VDFS) is a purpose-built distributed file system designed specifically to provide access for consumable protocols such as NFS. It is currently exposed through protocol containers that export SMB2, SMB3, NFSv3 or NFSv4.1 as a part of vSAN file services.

NFS Front end access

While VDFS does offer a cluster-wide namespace NFS4.1 will use the in-protocol redirection to support connections to a single namespace being re-directed out to the container that is hosting a given share. In this way, the primary IP used for the namespace does not proxy or hairpin the IO path, but acts as a redirection broker for client requests. This allows scaling out of throughput and performance as additional shares are assigned and balanced across different containers exposing the protocols in the cluster.

SMB front end access

Connecting to any container will present a base share path that is the namespace created for the cluster. DFS links will redirect to the specific container that a share has been placed on. Users can connect to any container where they will follow the DFS link to the specific host that is hosting a given share.

Round Robin DNS can additionally be configured to avoid the need to directly use the hostnames of the backing containers. In this example, vault.satm.eng.vmware.com has been configured as the domain namespace for the cluster along with forward DNS names that point to the IPs of the backing containers.

Expanding back end placement

VDFS is a distributed file system that sits inside the hypervisor directly on top of vSAN objects providing the block-based back end. As a result, it can easily consume SPBM policies on a per-share basis. New objects are automatically added and concatenated onto a VDFS volume when the maximum object size is reached (256GB). Components behind these objects can be striped, or as a result of various reasons be automatically spanned and created across the cluster. This allows shares to automatically "Scale deep" in their capacity usage while scaling out the performance load across additional back-end hosts, and devices.

vSAN data-in-transit encryption

Data-at-rest encryption was introduced in vSAN 6.6 making it the industry’s first native HCI security solution. vSphere 6.7 and vSAN 6.7 cryptographic modules achieved FIPS 140-2 validation by the National Institute of Standards and Technology (NIST), that specifies the security requirements for cryptographic modules.

vSAN 7 Update 1 further improved its security stance with the addition of data-in-transit encryption using the same FIPS 140-2 validated encryption module to secure vSAN data as it traverses the vSAN backend network. Data-in transit encryption can be used independently or in conjunction with data-at-rest encryption to achieve the desired level of protection.

File services in vSAN 7 U2 now also support Data-in-Transit encryption. In addition, for 2-node environments using a shared witness host appliance vSAN 7 U2 will support DiT encryption for the shared witness.

VDFS Management and Control Plane

Each node has a control path that helps manage responsibilities such as monitoring the health, and handling failover and load balancing. It also provides the conduit for connectivity to vCenter and the vSAN management daemons on each host. vCenter Includes management modules in the vsan-health daemon.. vCenter is not required for access to shares and does not sit within the I/O path. The VDFS controller services are distributed across ESXi hosts in the cluster. These services handle monitoring of the share and file system health and communicate with the vSAN management daemons, and remediate failures at the protocol endpoint layer above. This layer is what helps tolerate failures to an individual host, a VM, a network, or storage.

VDFS IO Path

VDFS sits above the vSAN layer, but below the virtual machine layer within the IO path. VDFS daemons run inside the hypervisor. They work closely with the vSAN services to pass IO, create and expand shares automatically, and monitor and recover from failures.

When setting up vSAN file services you will notice that there is only a single front facing network used to export NFS or SMB to clients. No additional backend networking to the vSAN is required, and no VMDKs are attached to the container hosts. Traditionally a file server running inside a virtual machine would need to use one of these methods and incur additional compute and networking overhead as data would have to travel over ethernet, or through the internal vSCSI layers. The container hosts bypass the traditional vNIC or vSCSI methods of connecting to the storage layer, but rather, uses an optimized transport protocol to reduce hops and improve efficiency. This connection speaks to the hypervisor layer where VDFS runs. It is used to transmit the data to a hypervisor-native VDFS. This provides load balancing through share volumes that can be accesses from any client in the cluster, performance through zero-copy, and ease of use through a simple connection of the file client to the SMB/NFS server (container). VDFS is also responsible for translating the commands into block requests, to the vSAN layers. This design optimizes performance, overhead and security of data access.

VDFS Write IO path

The data moving from the protocol services layer to the VDFS layer uses a form of Zero-Copy operation. This efficient alternative to a network hairpin, or trip through the vSCSI layer helps reduce latency and keep the compute overhead cost of file operations down.

Protocol Service Architecture

The front end file protocol services run in containers. This enables rapid recovery from failure, quick upgrades to the protocol stack, and a scale out architecture. The image for the containers and container hosts is included within the vSAN File Services OVA package. This package can be automatically downloaded and updated from within vCenter if it has an internet connection, or may be manually updated in an air gap environment. The file services images are cached on the vCenter server for fast re-deployment.

These containers hosts, displayed as FSVMs in vCenter, and managed entirely by the system, use a special Storage policy called: FSVM_Profile_DO_NOT_MODIFY. This storage policy is controlled by the file services, and is designed to help provide resilience of the services, not the data. Data level resilience is controlled by a single storage policy applied to the share. It is worth noting that vSAN File Services will replace an impacted container host, and is not dependent on vSAN policy to protect the objects backing them. This policy is NEVER used for the file shares and data protection of file shares is handled by the vSAN policy assigned to the share. The File shares do not use VMDKs for storage. The container hosts remain independent from the presentation of a discrete share, thus are pinned to an ESXi host through host affinity. This provides a platform for the protocol services rendered by the container to spawn on another host in the event of maintenance or a failure. As the name implies, this storage policy should not be modified, and it should only be assigned to the File Services agent VMs. The number of containers used for vSAN file services will be equal to the number of hosts in a vSAN cluster. Note that these agent FSVMs are deployed into an “ESX Agents” resource pool. They cannot be moved into another resource pool, as they are a non-manageable entity.

Monitoring of vSAN File Services

Performance metrics for file shares can be viewed on a per-share basis. These performance metrics are provided by the vSAN Performance Service, and are visible in the vCenter UI. The APIs in vSAN also allow for file share performance data to be viewed in other applications, such as vRealize Operations.
Read and write metrics for:

IOPS
Throughput
Latency

Skyline health checks for file services

Infrastructure Health
File Server Health
Share Health

Snapshots for vSAN File Shares

vSAN 7 Update 2 offers support for file share snapshots. A vSAN Distributed File System (VDFS) snapshot is a file system snapshot that provides a point-in-time image of a vSAN file share. When the vSAN File Service is enabled, you can create up to 32 read-only snapshots per share. VDFS snapshots can be managed via vSAN Management API, PowerCLI, or manually in the vCenter UI.

VDFS snapshots can be created and deleted from the vCenter UI, and managed by browsing to a hidden folder in the file share path. (\\file-server.com\file-share\.vdfs\snapshot).

vSAN File Share Snapshots in PowerCLI

Recently PowerCLI 12.3 announced support for vSAN file service snapshot management. There are 3 new cmdlets below that can be used to manage VDFS snapshots. These cmdlets can be used to create and present VDFS snapshots as a backup source. The snapshot script can then be invoked as a pre-job task to ensure that every backup job creates a new snapshot and uses it as the source for the backup.

New-VsanFileShareSnapshot

This cmdlet creates a vSAN file share snapshot for the specified vSAN file share.

1 #Creates a new vSAN File Share Snapshot
2 PS C:\> New-VsanFileShareSnapshot -Name "FileShareSnapshot1" -FileShare $fileShare

Get-VsanFileShareSnapshot

This cmdlet retrieves vSAN file share snapshots based on name or other filters.

1 #Retrieves all vSAN File Share Snapshots
2 PS C:\> Get-VsanFileShareSnapshot

Remove-VsanFileShareSnapshot

This cmdlet removes vSAN file share snapshot.

1 #Removes vSAN File Share Snapshot
2 PS C:\> Remove-VsanFileShareSnapshot -FileShareSnapshot $vSanSnapshot -Confirm:$false

Configure backup software to use VDFS snapshot

While backup vendors work to add vSAN file share snapshots into their software, organizations can use PowerCLI today to present them as a usable backup source. Using the VsanFileShare cmdlets, and backup vendor PowerShell modules create a script and configure it to run prior to the backup job. The script should accomplish the following:

Create a new file share snapshot (PowerCLI)
Delete all older file share snapshots (PowwerCLI)
In the backup software, point the file share backup source to the newly created snapshot. (vendor backup PowerShell module)

As an example, the following sample script creates a snapshot for a file share named "Profiles". It then uses Veaam Backup PowerShell to use the newly created snapshot as the backup source. As with all sample scripts, this is only an example and should be used at your own risk.

Space reclamation for file shares using UNMAP

In an attempt to be more efficient with storage space, modern guest OS filesystems have had the ability to reclaim no longer used space using what are known as TRIM/UNMAP commands for the respective ATA and SCSI protocols. Since vSAN has had full awareness of the TRIM/UNMAP commands since 6.7 U1. This is an opportunistic space efficiency feature that can deliver much better storage capacity utilization in vSAN environments. Now in vSAN 7 Update 2 VDFS reclaims or shrinks file shares as files are deleted and returns space to be used elsewhere on vSAN.

Improved scale, performance and efficiency

And finally, vSAN 7 U2 optimizes some of the metadata handling and data path for more efficient transactions, especially with small files. In this release, the maximum number of shares per cluster has been increased from 32 to 100. Additional optimizations to the underlying file system proxies helps reduce context switching, minimizing overheads and improving latency for file handling.

vSAN File services will show a warning alert when 200 simultaneous SMB connections are exceeded. While it will operate beyond this, it is advised to add more root shares, and balance the load more widely across the cluster.

Access Based Enumeration

With vSAN 7 U3, the vSAN file services introduces support for a technique of intelligent disclosure known as “access-based enumeration” or ABE. This feature, capable with file shares using SMB v3 and newer, reduces the disclosure of content to those with the appropriate permissions. Support of ABE will prevent a directory from listing all folders within a share, and only list those with the appropriate permissions for access. This type of intelligent disclosure helps prevent unintentional disclosure of information around sensitive material even though a user may not have full access to the material.

vSAN Express Storage Architecture (ESA) Support

VMware vSAN File Services is not yet supported with VMware vSAN Express Storage Architecture. VMware Original Storage Architecture is still required as of vSAN 8 Update 1.

Additional Information

The following contains additional links and presentations that may be useful for vSAN File Services. Note, a number of common questions can be found here in the vSAN FAQ.