vSAN performance questions - How do I identify if I have bad performance?

This question came in this morning.

𝘔𝘺 𝘕𝘝𝘔𝘦 𝘤𝘭𝘶𝘴𝘵𝘦𝘳 𝘪𝘴 𝘴𝘭𝘰𝘸, 𝘢𝘯𝘥 𝘐 𝘵𝘩𝘪𝘯𝘬 𝘪𝘵'𝘴 𝘣𝘦𝘤𝘢𝘶𝘴𝘦 𝘐 𝘯𝘦𝘦𝘥 𝘵𝘰 "𝘰𝘱𝘵𝘪𝘮𝘪𝘻𝘦 𝘵𝘩𝘦 𝘥𝘳𝘪𝘷𝘦𝘴, 𝘥𝘰 𝘺𝘰𝘶 𝘩𝘢𝘷𝘦 𝘢 𝘨𝘶𝘪𝘥𝘦 𝘧𝘰𝘳 𝘵𝘩𝘪𝘴?

From time to time I see various performance related questions and they often start with some common issues:

1. They fail to identify WHY they think there is a problem with the back end.

2. They fail to demonstrate understanding of what a performance problem looks like.

So rather than directly respond to this question, we will look at how to identify if a performance problem is not actually present (or not at the storage layer).

IOPS is not A unit of good or bad performance. It's a measure of use of a system (Specifically the number of operations per second). In a vacuum seeing this number low doesn't actually tell us anything for how good transactional application performance. Now anecdotally I can tell you seeing sub 50 IOPS there is not a large amount of workload being generated. Specifically at this low of a level I would call this "background noise". Though to fully qualify if the system is being stressed (or frankly used at all) and this number is a bottleneck we would want to look at corresponding latency at this layer. In this situation the cluster is demonstrating incredibly low latency (below 1ms).

HIt would appear the reason IOPS are low is simply because nothing is really trying to use the system. Now there could VERY much be a real application/end user problem, and even could be one manifested as latency at the virtual machine layer cause dby something above the disk layer.

What one can identify at this point is that the problem is not being caused by the back end disks being overtaxed, as if that was the case we would see back pressure/latency from them struggling to keep up.

Also of note, if the back end disk latency was high, and usage was low it could signal a problem at a lower level (Say firmware or even thermal throttling issue).

Next up we might ask the person who asked us this?

So why do you think it's slow?

Part of troubleshooting performance isn't just assuming it's the layer below you and blindly inventing a theory of why it's at fault. Thankfully we have some tooling to get better visibility to WHERE in the I/O path the latency is coming from. vSAN offer a free tool to map out all of the paths in the I/O for a VMDK and show which pieces are contributing to latency. In the below example latency is coming from the networking stack. In this example you may wish to make sure links are sized correctly, there are no errors caused by cable issues etc.

docs.vmware.com/en/VMware-vSph…

The key issue at hand in reviewing this cluster is understanding that that Latency not IOPS or throughput is how we qualify that a layer is being stressed or having an issue.

Should I measure performance usage in throughput or IOPS?

Higher throughput and lower IOPS (or the inverse) isn't bad. It just means generally a function of the block sized used. Remember: IOPS * Block Size = Throughput!

At first glance you may think the test of the left "slower" for pushing less than half the IOPS of the cluster on the right. A closer inspection reviews though the cluster on the right is pushing 7x the amount of data from disk beacuse it is using a larger block size.

Which test should I use for testing my cluster?

It is benefitial to run both benchmarks, and understand that throughput tends to tax some components more (Networking layers), while small block IOPS focused benchmarks tend to tax other layers (CPU). Different applications and uses of those applications may stress throughput or small block IOPS more.

How do I know what my apps/users are using?

While some application vendors offer general guidance on their block size usage, they may not truely reflect all uses of the application or how the data is coalcesed and sent to disk by the operating system and in guest file system. VMware vSAN I/O Insight will actually let you capture this (and a ton of other stuff). Here is a short demo video duncan shows that includes graphs for block size as well as read/write mix. Remember writes are commonly more difficult to process than reads by storage systems.

You can schedule vSAN I/O Insight to run at different times, as you might have completely different patterns when overnight reporting runs, or backups run, or DB maintenance is performed.

This has been your #VMwareMailbag thread of the week.

Troubleshooting vSAN Performance | VMware This document provides a framework for the diagnosis and mitigation of performance issues in a production vSAN environment It reflect features and capabilities up to and including vSAN 7 U3. https://core.vmware.com/resource/troubleshooting-vsan-performance

vSAN performance questions - How do I identify if I have bad performance?

Should I measure performance usage in throughput or IOPS?

Which test should I use for testing my cluster?

How do I know what my apps/users are using?

Filter Tags