Optimized ESXi CPU Scheduler for AMD EPYC Processors
Introduction
vSphere 7 Update 2 release comes with a highly optimized ESXi CPU scheduler for AMD EPYC processors.
With AMD EPYC processors arrival, we felt that there is an opportunity to fine tune the ESXi CPU scheduler to ensure that virtual workloads are evenly distributed across Last Level Cache (LLC) domains per Non-Uniform Memory Access (NUMA) node. We noticed that in some instances, more vCPUs were scheduled than available core in an LLC domain. This sometimes resulted in suboptimal performance.
The optimized ESXi CPU scheduler is fully AMD EPYC architecture-aware or Core Complex (CCX)-aware and provides enhanced load balancing and improved cache locality.
Under the Hood
As more workloads run on AMD EPYC platforms we’ve seen the uneven distribution of workload can be remediated through enhanced coordination between the NUMA scheduler and CPU scheduler.
When a VM is powered on, the NUMA scheduler breaks the VM into one or more NUMA clients and assign each client a NUMA home node. The responsibility of the NUMA scheduler is to optimize NUMA client's home node selection. It then delegates the Physical CPU placement within a NUMA node to CPU scheduler assuming CPU scheduler will produce optimal placement/load balancing within a NUMA node.
However, the CPU scheduler mostly looks within the LLC boundaries while making load balancing decisions, instead of NUMA node. As a result, when there are multiple LLCs per NUMA node, neither CPU scheduler nor NUMA scheduler is actually looking for correct placement across LLCs but within a single NUMA, resulting in scheduling more cores than available.
After finding the issue, we acted swiftly and introduced a new topology flag, "multiLLCsPerNUMANode." This topology flag handles capturing whether the system has multiple LLCs per NUMA node or not and is set automatically on AMD EPYC platforms.
With the introduction of this flag and optimization of the CPU scheduler, we have observed :
- Double-digit performance gains in VMmark and higher VM density
- Significant performance gains in vSphere with Tanzu & Tanzu Kubernetes Grid Cluster
- Enhanced performance in critical database workloads
The optimized ESXi scheduler is now capable of running workloads with optimal performance on AMD EPYC platforms.
This white paper contains detailed information about ESXi CPU scheduler optimization for AMD EPYC processors.
Conclusion
VMware is committed to making customers' lives as easy as possible. The recent ESXi CPU scheduler optimization ensures that customers can run their workload on AMD EPYC processors without worrying about sub-optimal performance.