vSAN Performance Evaluation Checklist

Note that performance will not be as good as RAID-1, due to overheads such as parity calculations and Read-Modify-Write operations for partial writes.

Note that performance will not be as good as RAID-1, due to overheads such as parity calculations and Read-Modify-Write operations for partial writes.

Checksum has a performance impact for write workloads. This is due to the overhead of checksum calculations and extra checksum IO to disk.

Enabling Deduplication and Compression will cause additional IOPS and latency overhead, especially on write heavy workload. This is mainly due to metadata IO overhead.

Recommendation: Only enable encryption if there is hardware assisted encryption support such as Intel's AES-NI.

Enabling Encryption increases the CPU cost per IO increases because of data encryption overhead.

• Decide on number of VMs per host. Initial recommendation is to deploy 2 VMs per diskgroup.

• Decide on number of VMDKs per VM (e.g. 8 which is default )

• Decide on size of VMDK (e.g. 10GB, which is default)

• Decide on Outstanding IO (OIO) per VM (e.g. 2 to 4). VMware recommends 4 OIO per VMDK. If resulting latency is too high, OIO can be lowered.

• Decide on Block Size (e.g. 4K) Smaller blocks sizes give better IOPS results on vSAN, but larger block sizes can give better throughput.

• Decide on Read/Write Ratio (e.g. 70/30)

• Decide on Random or Sequential IO. Random IOs give better performance on vSAN.

Note: The prepare step will take a considerable amount of time as each VM disk will have data written to it in a sequential fashion to ensure we do not hit a first write penalty.

The workload is set to 70% Read, 100% Random and Outstanding IO is set to 4 threads per VMDK. There is a 30 minute warmup period, following by 60 minutes tests (results are only based on the 60 minutes testing).

There are two considerations with this setting:

1. Clear the cache: the reason for clearing the caches is to start from a blank slate with each test – removing the effect of a prior test (except for the actual data stored on the capacity disks) and therefore increasing the repeat-ability of test results. This is a best practice for comparing performance between different test configurations, e.g. RAID-1 vs RAID-5, dedupe enabled vs dedup disabled, etc. Though we recommend clearing caches for repeat-ability and isolation between tests, this will increase the amount of soak-time needed to achieve run a benchmark test due to the fact that cache needs to be repopulate to achieve optimal performance conditions.
2. Don't clear the cache: Clearing the cache on hybrid vSAN causes a drop in read performance until the read cache is re-warmed. Running a test for a long duration will effectively push out the old contents of both the read and write cache, accomplishing the same goal flushing the caches and then re-filled them with the new contents. Keeping the caches intact will show more realistic and consistent performance throughout testing. Thus if the goal is to gather "steady state results" or benchmark against a competing product, the recommendation is not to clear the cache to achieve optimal performance conditions more quickly.

If you wish to clear the cache, this feature requires that SSH is enabled on all hosts, and that the ESXi username and password fields are populated.

• Increment as you test, until you find your sweet spot between IOPS, throughput and latency. Alternatively, use 'Number of Threads Per Disk' to achieve the goal.
• Rinse and repeat

• Avoid tuning the number of VMDK as you will need to initialize the disks with each new test. Outstanding IO can be tuned via Number of Threads per Disk.

• Increment as you test, until you find your sweet spot between IOPS, throughput and latency. Alternatively, use 'Number of VMs' to achieve the goal.
• Start with a small value of 1 or 2, and gradually increment. This will give you a balance for IOPS, Throughput and Latency that you can fine tune.
• Rinse and repeat

• Pick a few common block sizes, for example 4KB, 16K and 64K. Note that very large blocks are chunked by vSAN to 64KB, resulting in IO amplification and triggering congestion. Smaller blocks should be used for benchmarking.
• Rinse and repeat

• This will avoid having to reinitialize the data in the VMDKs with each run.
• This will also ensure consistency across tests as you are using the same data with each iteration.

• This will avoid having to reinitialize the data in the VMDKs with each run.
• This will also ensure consistency across tests as you are using the same data with each iteration.

• This may take a long time to complete. DO NOT SKIP THIS TEST, or your performance results will be sub optimal
• If you decide to enable/disable dedupe during your testing, you will need to choose a different prepare option
• If you are re-using the VMs, you do not need to repeat this step

Length depends on cache size, incoming rate of writes, and write cache drain rate to capacity tier (which can vary with disk group configuration and features)
 

• Objective is to capture performance while cache is being utilized and destaging from cache tier to capacity tier is occurring.
• While the test workload is running, look at the graph for "Write Buffer Free Percentage". If the percentage of write buffer free space is decreasing, then vSAN is taking in writes faster than it is processing them to their final home in the Capacity tier. You should continue to run the workload until the Write Buffer Free Percent stays the same or increases for a 30 minute period.
• Write Buffer Free and Cache Disk Destage rate can be viewed in the vSAN Performance Graphs via Hosts > Monitor > Performance > vSAN Disk group

• Deduplication and Compress will add significant overheads with large block, random workloads
• Checksum will add significant overheads with large blocks (>64KB)
• Erasure Coding Policies (R5/R6) will introduce IO amplification on  ALL writes, but partial writes (which involve a read-modify-write operation) will introduce considerably more IO amplification.
• Recommendation: Enable new data services or introduce policy changes one at a time. Don't change lots of things at once. Make sure that the current set of test VMs are removed before making any policy and/or data service changes, as this will add unnecessary time to the performance test

Performance Diagnostics is a built-in utility which can provide guidance towards achieving better benchmark results.
This requires CEIP (Customer Experience Improvement Program) to be enabled.
This is integrated with HCIBench User Guide and the vSAN Support Insight documentation https://core.vmware.com/resource/vsan-support-insight