Using Fault Domains in vSAN ESA

The post "Design and Operation Considerations When Using vSAN Fault Domains" helped many of our customers understand the basic principles of vSAN's optional Fault Domains feature, as well as offering some guidance in the design and operation of a cluster using this capability. 

But how do these concepts change in a cluster running vSAN's Express Storage Architecture, or ESA?  Let's look at some of the similarities and differences when using this feature in a vSAN ESA cluster as opposed to a vSAN cluster using the Original Storage Architecture (OSA), and why you might want to consider moving to the ESA if you use or are interested in vSAN's Fault Domain feature. 

Benefits of using Fault Domains in ESA versus OSA

The concept of fault domains in the ESA has remained the same as they were in the OSA.  When this optional feature is used, an Administrator will define a set of fault domains where each fault domain represents a physical boundary of failure, such as a rack, or a data closet.  vSAN does the rest by spreading the data across the fault domains in a resilient way to ensure availability upon a failure.  Figure 1 shows the configuration of the Fault Domains feature in the vSphere Client.

Figure 1.  A vSAN cluster with 32 hosts spread across 8 racks using the Fault Domains feature.

But the ESA can deliver performance and efficiency in ways that are not achievable with the OSA.  This most certainly extends to topologies that use the vSAN Fault Domains feature. 

Recommendation.  Ensure that a cluster using vSAN Fault Domains meets the proper network requirements.  The Fault Domains feature must not exceed 1ms RTT of network latency between hosts and fault domains. vSAN ESA has higher bandwidth requirements to ensure the network can deliver the performance capabilities of the hosts.

Space Efficiency

Since the ESA can store data in a space-efficient RAID-5/6 erasure coding with the performance of RAID-1 mirroring, you can have highly efficient rack-level resilience with no compromise in performance.  Let's look at a simple comparison between two 24-host clusters.  Both are using vSAN's Fault Domains feature and assume that the data will be resilient to protect against rack-level failure, with an FTT of 1.

On the left side of the illustration, we see a cluster running the OSA using 6 fault domains with 4 hosts per fault domain.  It was common for customers to use RAID-1 mirroring in these cases to ensure optimal performance.  But this means that for every 100GB of stored data, it would consume about 200GB of raw capacity.  In comparison, on the right side of the illustration, we see a cluster running ESA using 6 fault domains with 4 hosts per fault domain.  The same rack-level resilience (FTT=1) can be achieved but with much better space efficiency and performance that meets or exceeds RAID-1.  Using ESA's RAID-5 storage policy with 6 defined fault domains, every 100GB of stored data will be consuming just 125GB of raw capacity.  That is a 37.5% savings in capacity over the similar cluster running the OSA but with much better performance!

Figure 2.  Comparing capacity efficiency between OSA and ESA when using the Fault Domains feature.

Some customers have chosen to use RAID-1 mirroring with the Fault Domains feature in the OSA because it required the fewest amount of fault domains (3 fault domains being the absolute minimum, and 4 being the recommended number).  The ESA uses a unique adaptive erasure coding scheme for RAID-5 that will use one of two RAID-5 data placement schemes depending on the cluster size.  When using the Fault Domains feature in the ESA, RAID-5 erasure coding can be used with as few as 3 fault domains, but 4 fault domains being the recommended minimum.  In these configurations, every 100GB of stored data will be consuming just 150GB of raw capacity.  That is still a 25% savings in capacity over a similar cluster running the OSA but with much better performance.

Reduced Traffic across the Network Spine

Standard vSAN clusters not using vSAN's disaggregation capabilities or vSAN's Fault Domains feature typically only use the network resources of the Top of Rack (ToR) leaf switches in a spine and leaf network topology.   Larger clusters may introduce traffic across the spine only because the host count exceeds the physical resources of the rack or ToR switches. 

When using vSAN's Fault Domain feature for rack-level resilience, the feature will introduce network traffic across the spine because it must use the spine switches to traverse vSAN traffic from one set of ToR switches in one rack to ToR switches in other racks.  This change in traffic may not have been considered during the initial capacity planning and design of the network. 

The ESA in vSAN can help reduce I/O payload sizes across any network switching, whether it be ToR leaf switches, or spine switches because of the ESA's ability to compress the data prior to any transmission across the network.  The result will be less data transmitted across the network for each guest I/O processed.

Figure 3.  Reduced traffic across the network through compression in vSAN ESA.

The benefit of compression with vSAN’s Fault Domain feature is very similar in how compression can improve stretched cluster topologies, described in the post, "vSAN ESA in a Stretched Cluster Topology."

Design Guidance

The design guidance of a cluster using the fault domains feature remains very similar in an ESA cluster as it does an OSA cluster. 

The number of fault domains recommended in a cluster.  We still recommend having at least one more fault domain than the minimum required for the storage policy you are using.  For example, if you are planning on using RAID-6 for a highly resilient FTT=2, the minimum number of fault domains for RAID-6 is 6, but you will want to have at least 7 fault domains to ensure a spare fault domain is available for regaining the prescribed level of resilience in the event of a sustained rack failure.  The number of fault domains recommended in a cluster using RAID-5 in the ESA will depend on the desired goal.  RAID-5 running in its smaller 2+1 stripe with parity can run on as few as 3 fault domains, but as noted above, we recommend having at least 4 fault domains.  To take advantage of vSAN ESA’s more space efficient RAID-5 using a 4+1 stripe with parity, you must have at least 6 fault domains.  Even though it spreads the data with parity across 5 fault domains, the ESA will automatically change it to the 2+1 scheme if there are not at least 6 fault domains. For more information, see the post “Adaptive RAID-5 Erasure Coding with the ESA in vSAN 8.”

The number of hosts recommended within a fault domain.  While the technical minimum is a single host per defined fault domain, the post "Design and Operation Considerations when using vSAN Fault Domains" recommends at least 3 hosts per fault domain for various reasons.  This is applicable to the ESA just as it is to the OSA.

Fault domain symmetry.  Just as with hosts in a standard vSAN cluster, there is no strict requirement for resource symmetry.  But it is recommended to have reasonable levels of symmetry (in host count, CPU, memory, and storage capacity).  For more information, see the post:  "Asymmetrical vSAN Clusters - What is Allowed, and What is Smart."

Overall cluster size.  Since the fault domains feature within vSAN is designed to provide rack or room-level resilience in the event of a failure, it is really intended for larger clusters.  Given the recommendations of having at least 3 hosts within each fault domain, and one more fault domain than is absolutely required by the desired storage policy, a realistic minimum host count would look like the following:

• FTT=1 using RAID-5 (using its 2+1 scheme plus one extra fault domain) is 12 hosts: 4 fault domains with 3 hosts in each. 
• FTT=1 using RAID-5 (using its more space efficient 4+1 scheme plus one extra fault domain) is 18 hosts:  6 fault domains with 3 hosts in each
• FTT=2 using RAID-6 (using its 4+2 scheme plus one extra fault domain) is 21 hosts:  7 fault domains with 3 hosts in each.

Limitations

Just as with the OSA, there are a few features that are unavailable or unsupported when using vSAN's Fault Domain feature.

• Reserved Capacity toggles.  As noted in the post, "Understanding Reserved Capacity Concepts in vSAN" and "Design and Operation Considerations when using vSAN Fault Domains" when a cluster (ESA or OSA) is configured with the fault domains feature, the "Operations Reserve" and "Host Rebuild Reserve" toggles are not available.  It is recommended to follow free capacity recommendations in vSAN prior to those capabilities coming out, which was to maintain about 25% of free capacity.
• vSAN ESA Auto-Policy Management feature.  The new Auto-Policy Management Capabilities with the ESA in vSAN 8 U1 are currently not supported when using the fault domains feature in vSAN ESA.

Remember that it is not possible to configure a cluster using vSAN's Fault Domains feature with just 2 fault domains.  You must have at least 3 fault domains, with the recommended number of fault domains being more depending on which RAID data placement scheme is used.

Summary

Using the Fault Domains feature in a cluster running vSAN's Express Storage Architecture allows you to provide unique resilience capabilities that align with your physical topology.  But the architecture of the ESA delivers this type of resilience with space efficiency and performance that is not possible with vSAN's Original Storage Architecture.

@vmpete