vSAN 7 Update 2 RDMA Support

April 01, 2021

Networking Matters

The nature of a distributed storage system like vSAN means that a network connecting the hosts is heavily relied upon for resilient connectivity, performance, and efficiency. Traditionally, vSAN has relied upon TCP communication to move data between hosts in a reliable and consistent manner. vSAN 7 Update 2 now supports clusters configured for RDMA-based (Remote Direct Memory Access) networking. Specifically, RoCE v2 (RDMA over Converged Ethernet version 2) will be supported. Transmitting native vSAN protocols directly over RDMA can offer a level of efficiency that is difficult to achieve with traditional TCP-based connectivity over ethernet. With vSAN 7 Update 2, vSAN communications and payload will be able to use RDMA, which (on its own) will offer low levels of latency, high throughput, and high IOPS, all with reduced CPU utilization per I/O. While this is just one factor that contributes to performance and efficiency, RDMA can help position the network to not be the primary bottleneck.

vSAN RDMA

RDMA provides direct memory access from the memory of one computer to the memory of another computer without involving the operating system or CPU. Transmitting native vSAN protocols directly over RDMA can offer a level of efficiency that is difficult to achieve with traditional TCP-based connectivity over ethernet. This results in lowered CPU overhead and improved performance for sequential reads and random mixed read/write workloads.

Please consult with your switch vendor for identifying switches that will support RCoEv2. vSAN RDMA Ready network adapters can be found on the vSAN VCG.

vSAN 7 Update 2 RDMA

vSAN RDMA is enabled with a single toggle. If all hosts cannot establish an RDMA connection, vSAN will automatically revert the configuration for the cluster to TCP networking.

RDMA network vSAN 7 Update 2

NIC Teaming with RDMA

RDMA for vSAN supports the following teaming policies for virtual switches:
• Route based on originating virtual port
• Route based on source MAC hash
• Use explicit failover order

vSAN 7 Update 2 RDMA interface

RDMA Health Checks

RDMA skyline

Additional health checks will be included to provide visibility into RDMA if it is enabled in the cluster. These checks will make sure the network is configured for lossless traffic using data center bridging (DCB) and that priority flow control (PFC) is configured for a value of 3.

Host level network monitoring has increased. Within the vSAN performance service, new metrics and default alarms have been added to 7 Update 2.

RDMA alarm

For additional questions about this topic, feel free to reach out to the author John Nicholson on Twitter:  @Lost_Signal

Additional links:

vSAN Quick Links

vSAN VCG for RDMA Network Adapters

Filter Tags

Storage vSAN vSAN 7 Blog What's New Intermediate