SAP HANA on VMware vSphere Best Practices and Reference Architecture Guide

Introduction

Abstract

This guide is the 2024 July edition of the best practices and recommendations for SAP HANA on VMware Cloud Foundation, focusing on vSphere. It describes the best practices and recommendations for configuring, deploying, and optimizing SAP HANA scaled-up and scaled-out deployments, with a focus on vSphere 8.0 running on 3rd- and 4th-generation Intel Xeon Scalable processors, such as Cascade Lake, Cooper Lake, Ice Lake, and Sapphire Rapids systems.

Most of the guidance provided here results from ongoing joint testing by VMware and SAP to characterize the performance of SAP HANA running on vSphere.

vSphere 7.0 and vSphere 7.0 update versions on second and third-generation Intel Xeon Scalable processors, such as Broadwell, Skylake, Cascade Lake, Cooper Lake, and Ice Lake ESXi hosts are still supported and are covered in this document as well.

We also provide information on Intel Optane Persistent Memory (PMem) 100 series technology, which is only supported with Cascade Lake and vSphere 7.0 virtualized SAP systems. SAP HANA does not support later CPU generations with PMem, and Intel has announced it is discontinuing this technology.

For earlier vSphere versions or CPU generations, please refer to the 2017 or 2022 edition of this guide.

Audience

This guide is intended for SAP and VMware hardware partners, cloud services providers, system integrators, architects, and administrators responsible for configuring, deploying, and operating the SAP HANA platform in a VMware virtualization environment.

It assumes you have a basic knowledge of VMware Cloud Foundation concepts and features, SAP HANA, and related SAP products and technologies.

Solution overview

SAP HANA on VMware vSphere

Per SAP, 80% of their German and over 51% of their US customers run SAP on-premises or in a private cloud. According to an IDC study in the same SAP article, up to 68% of SAP workloads will stay on-premises in the United States, especially for large customers. SAP has supported vSphere for production use cases for 10 years, and most of the on-premises customers (over 70%) use VMware solutions as their private cloud (SDDC) solution for SAP applications like SAP HANA.

By continuing to validate SAP HANA on the latest Intel CPU generations and latest vSphere and Cloud Foundation (VCF) versions, all these customers can continue to seamlessly integrate their IT and SAP operations by leveraging their existing IT processes, know-how, and customer-tailored infrastructure that allows full sovereignty over their data and the overall SAP HANA solution, which supports scaling database sizes up to 12 TB (16 TB is planned with 8-socket Sapphire Rapids systems) and scaling out with up to 16 nodes (plus high availability nodes) with up to 48 TB (depending on the host configuration) running on a vSphere virtualized infrastructure.

The SAP HANA platform with VMware virtualized infrastructure provides an optimal environment for achieving a unique, secure, cost-effective solution. It also offers benefits that physical deployments of SAP HANA do not include, such as:

• On-premises security and control
• Locality for consistent and predictable performance
• Regulatory demand for cloud neutrality
• Sovereignty over data and business transactions
• Increased security (using VMware NSX as a zero-trust platform)
• Higher service-level agreements (SLAs) by leveraging vSphere vMotion to migrate live SAP HANA instances to other ESXi host systems before hardware maintenance or host resource constraints
• Integrated lifecycle management provided by VMware Cloud Foundation SDDC Manager
• Standardized high availability solution based on vSphere HA
• Built-in multi-tenancy support via SAP HANA system encapsulation in a VM
• Easier hardware upgrades or migrations due to the abstraction of the hardware layer
• Higher hardware utilization rates
• Automation, standardization, and streamlining of IT operations, processes, and tasks
• Public cloud operating model and cloud readiness due to software-defined data center (SDDC) SAP HANA deployments

These and other advanced features found almost exclusively in virtualization lower the total cost of ownership and ensure the best operational performance and availability. As mentioned in SAP Notes 2937606, 3102813, and 3372365, and SAP KB 2101244, this environment fully supports SAP HANA and related software in production environments, as well as SAP HANA features such as multi-tenant database containers (MDC) and system replication (HSR).

Note: Refer to SAP Note 2104291 "FAQ - SAP HANA multi-tenant database containers," page 2.

Solution components

An SAP HANA system based on VMware technologies is a fully virtualized and cloud-ready infrastructure solution running on VMware vSphere and supporting technologies, such as VMware vCenter. All local server host resources—such as CPU, memory, local storage, and networking components—are presented to a VM virtually, abstracting the underlying hardware resources.

The solution consists of the following components:

• VMware-certified server systems as listed in the VMware hardware compatibility list (HCL)
• SAP HANA-supported server systems, as listed in the SAP HANA HCL
• SAP HANA-certified hyperconverged infrastructure (HCI) solutions, as listed in the SAP HANA HCI HCL
• VMware Cloud Foundation or vSphere Foundation with VMware products like:
  • vSphere and vSAN version 7.0 U2 and later, Sapphire Rapids-based systems, vSphere 8.0 and vSAN U2 and later
  • vCenter 7.0 and later, vCenter 8.0 with vSphere 8.0
  • Optional: NSX Networking and Security (out of scope for this document)
  •  Optional: Aria Suite for management (out of scope for this document)
• A VMware-specific and SAP-integrated support process

What's new in vSphere 8.0?

From “Introducing vSphere 8: The Enterprise Workload Platform,” VMware vSphere 8, the enterprise workload platform, brings the benefits of the cloud to on-premises workloads, supercharges performance through DPUs and GPUs, and accelerates innovation with an enterprise-ready integrated Kubernetes runtime. Additionally, there are significant operational benefits, like VMware Cloud Disaster Recovery add-on services, VMware Aria, a unified multi-cloud management solution that provides capacity planning and optimization for your infrastructure with the right size to fit the current and future needs of your SAP workloads, host different GPU workloads on a single GPU, and perform pre-staged ESXi upgrades, to name some. Visit the VMware by Broadcom specific product webpages for details and up-to-date information.

Benefits of SAP HANA on VMware vSphere 8.0

The latest advancements in vSphere 8.0 bring significant benefits to SAP customers, facilitating the creation of robust, cost-efficient, manageable, and high-performing SAP HANA environments. A notable enhancement for SAP users in vSphere 8.0 is the revamped approach to presenting the physical system/processor topology to virtual machines (VMs).

Previously, vSphere administrators had to manually configure SAP HANA VMs to align with the underlying host hardware, including non-uniform memory access (NUMA) alignment. This alignment is crucial for optimizing SAP HANA performance. To achieve optimal performance before vSphere 8.0, admins had to configure the advanced parameters for each VM.

The introduction of the enhanced virtual topology feature in vSphere 8.0 marks a significant improvement. This feature automatically determines optimal coresPerSocket values and virtual L3 cache sizes for VMs, simplifying configuration and enhancing performance.

vSphere 8.0 also incorporates intelligent, adaptive NUMA scheduling and memory placement policies, eliminating the need for manual VM balancing across nodes. Admins can still use manual controls to override the default behaviors and configure the NUMA placement for performance-critical SAP HANA VMs.

To learn more about optimizing performance, refer to the "Performance Best Practices for VMware vSphere 8.0" paper. For detailed configuration steps, consult the "VMware vSphere 8.0 Virtual Topology" paper.

In addition, vSphere 8 Update 1 marks the first release supporting up to 960 logical CPUs per physical host, further expanding scalability.

vSphere 8 Enterprise is available as part of VMware Cloud Foundation and VMware vSphere Foundation, as well as standalone editions such as vSphere 8 Standard and Essentials.   

Software and hardware support for SAP HANA on vSphere

SAP HANA production support for vSphere and VMware Cloud Foundation

In November 2012, SAP announced initial support for scaling up SAP HANA systems on vSphere 5.1 for non-production environments. Since then, SAP has extended its production-level support for scale-up and scale-out SAP HANA deployment options and multi-VM and half-socket support. vSphere versions 5.x and 6.x are no longer supported, and vSphere 7.0 will be unsupported in April 2025. Therefore, you should plan to upgrade to vSphere versions 7.0 or 8.0. The following table lists relevant SAP HANA on vSphere support notes as of March 2024.

Table 1. Relevant SAP notes

vSphere version support for Intel CPU platforms

The following table provides an overview, as of March 2024, of the vSphere 7/8 versions and CPU platforms supported by the SAP HANA on vSphere solution.

Table 2. Supported vSphere versions and CPU platforms as of March 2024

vSphere maximums

The following table summarizes the critical maximums of the different vSphere versions supported for SAP HANA. For more information, refer to the SAP HANA on vSphere scalability and VM sizes section.

Table 3. vSphere memory and CPU SAP HANA relevant maximums per CPU generation as defined by SAP

Note: These configurations may vary if smaller core count CPUs or different memory configurations are used and require an SAP HANA TDI/workload-based sizing. >8 socket Sapphire Rapids systems are in SAP HANA vSphere validation.

Release strategy

VMware’s SAP HANA certification and support strategies for vSphere are to support a single CPU generation or chipset with two versions of the hypervisor and to have a single hypervisor version span two CPU generations/chipsets. 

VMware does its best to balance supporting new customers on the latest hardware and those on an older platform. You may still use vSphere 7 since this version is supported until April 2025. vSphere 8.0 Update 3 is the most recent available version, and we recommend using this version for all new SAP HANA deployments on vSphere. The end of general support for this version is April 2027. For more details, refer to the VMware Product Lifecycle Matrix.
 

To directly show the ESXi versions: 

1. Set a filter for ESXi at the Product Release table heading, as shown in the following figure.

Figure 1. Select the filter icon next to Product Release and type ESXi

Supported vSphere and VMware Cloud Foundation offerings

SAP HANA on vSphere is supported only with the following editions:

• VMware Cloud Foundation
• vSphere Foundation

vSphere Standard can also be used for smaller SAP application deployments. vSphere Essentials Plus supports Business One solutions. Make sure you are familiar with the feature, CPU, and host limitations of these products.

Notes:

• We highly recommend VMware Cloud Foundation with bundled Select Support for strategic enterprise customers.
• Per SAP Note 2652670: SAP HANA VM on VMware vSphere, usually all update and maintenance versions of vSphere hypervisors are automatically validated within the same boundary conditions.
• This architecture reference guide and the involved test environment use only vSphere dvPortGroups to connect the SAP HANA nodes and non-SAP HANA DB / VM workloads. An upcoming version of this guide will include NSX-based vLANs or overlay segments.

References:

• The VMware vSphere Compute Virtualization white paper provides an overview of the licensing, pricing, and packaging for vSphere. For VMware support offerings, see the "On-Premises Support" section of the VMware Support Offerings and Services page. 
• For the latest SAP-released support information, refer to the SAP notes related to VMware or the SAP on the vSphere community wiki page.

Scalability and VM sizes

SAP HANA on vSphere is supported on the smallest SAP HANA system, which is a half-socket CPU configuration with a minimum of 8 physical CPU cores and 128 GB of RAM, up to 8-socket large SAP HANA VMs with up to 12 TB of RAM. The CPU power and RAM required for SAP workloads on HANA must be sized appropriately. VMs larger than 4 sockets require an 8-socket host. 8-socket servers are an optimal consolidation platform and could, for instance, host two large 4-socket VMs, each with up to 6TB of memory, 16 half-socket VMs with up to 750GB memory, or one large 12TB SAP HANA VM.

SAP supports VMware virtualized OLTP workloads up to 8TB for a 4-socket large SAP HANA VM and 12TB for an 8-socket large VM as standard sizes when selecting top-bin Intel CPUs. This standard sizing supports 50 % of the memory defined for OLTP-type workloads (for example, 3TB for 4-socket OLAP VMs or 6TB for 8-socket VMs). If additional memory is needed, you will need to do a workload-based SAP expert sizing.

For more details, review SAP Note 2779240: Workload-based sizing for virtualized environments. Table 4 shows the current vSphere maximums per physical ESXi host.

Table 4. vSphere physical host maximums (extract)

Note: ESXi hosts with up to 8 physical CPUs are supported. If larger 8-socket systems are required, contact your SAP or VMware account team and discuss deployment alternatives, such as scale-out or memory-tier solutions. Also note the support limitations when using 8-socket or larger hosts with node controllers (also known as glued-architecture systems or partially QPI meshed systems). 

The following table shows the maximum size of a vSphere VM and some other relevant parameters, such as virtual disk size and the number of virtual NICs per VM. These VM limits are higher than the SAP HANA–supported configurations. 

Table 5. vSphere guest VM maximums (extract)

[1] Review the Hardware Features Available with Virtual Machine Compatibility Settings web page for a detailed list of the guest hardware capacities. You must use hardware version 21 for VMs on (or migrated to) Sapphire Rapids hosts.

Deployment options and considerations

Reference architecture diagram

The following figure shows an overview of a typical VMware software-defined data center (SDDC) for SAP applications. At the center of a VMware SDDC is VMware Cloud Foundation, which includes vSphere, vSAN, and NSX. VCF has VI Workload Domains—each Workload Domain is a logical unit of application-ready infrastructure that groups ESXi hosts managed by a vCenter Server instance with specific characteristics according to VMware recommended practices.  

The figure shows the preferred VCF design for non-SAP HANA database VMs and SAP HANA with separate VCF VI Workload Domains. (You can include an optional, isolated VI Workload Domain.) Each VI Workload Domain has a dedicated NSX Manager cluster to provide the benefits of flexible lifecycle management, which includes the option to run different VCF software stack versions across the different VI Workload Domains. A VI Workload Domain can consist of one or more vSphere clusters provisioned by the SDDC Manager.

The red and blue VI Workload Domains are examples of SAP HANA Workload Domains with different storage types. The grey VI Workload Domain is an example of mixed SAP and non-SAP application Workload Domains with different SLAs/requirements, like ESXi hosts that are not SAP HANA-certified.

Figure 2. VMware Cloud Foundation for SAP applications

The next table provides an overview of the different Workload Domain types (WLDs) and how best to leverage these for SAP and non-SAP VMs/applications.

A VI Workload Domain (VI WLD) shares a vCenter single sign-on domain and the identity provider configuration with the Management Domain. You can manage all VI WLDs through a single pane of glass.

A VI WLD can share an NSX Manager instance with other VI WLDs. We recommend a dedicated NSX Manager per VI WLD for easier and more flexible lifecycle management.

Small and medium SAP deployments could consolidate SAP HANA and non-SAP HANA database VM workloads into a single VCF VI WLD with one or more vSphere cluster configurations to reduce the required hardware footprint, but this would lose the lifecycle management flexibility between different SAP workloads.

Larger deployments with SAP HANA systems with different operation, hardware, and SLA requirements should not share a VI WLD/cluster with non-SAP HANA VMs. Running these VMs in a dedicated SAP HANA VI WLD/cluster eases lifecycle management and ensures the correct cluster-wide settings for features like HA, DRS, and EVC.

An isolated VI WLD is another option for a distinct vCenter single sign-on domain; this requires an identity provider configuration. VI WLDs of this type cannot yet share an NSX Manager instance with other VI WLDs and, therefore, need a dedicated NSX Manager instance. This allows the independent lifecycle management of isolated VI WLDs and is recommended when strict isolation of user and business data is required.

Small SAP deployments could leverage the consolidated VCF architecture model, which consolidates the VCF Management components (vCenter, NSX Manager, and SDDC Manager) and all SAP workloads (non-SAP HANA database VMs and SAP HANA) with the lowest possible hardware footprint, but this loses the lifecycle management flexibility between the VCF Management Domain and the VI WLD running the different SAP workloads. If you want to deploy SAP HANA production-level VMs in a consolidated WLD, you must ensure these SAP HANA VMs run on SAP HANA TDI-supported systems and do not share a NUMA node with a non-SAP HANA VM. We advise configuring dedicated vSphere clusters to manage VCF VMs and workload VMs. For a consolidated WLD, we recommend separate cluster-wide settings for management and SAP NetWeaver/SAP HANA hosts.

Note: Every vSphere cluster requires at least 2 embedded vCLS VMs or 3 external vCLS VMs. A cluster allows different cluster-wide settings; however, we recommend you add non-SAP HANA hosts to an SAP HANA cluster to offload the vCLS VMs. 

Table 6. VMware Cloud Foundation Management and VI Workload Domain types and use cases

VCF Management Domain
First domain deployed, runs on dedicated ESXi hosts  (min. 4) Contains the following management appliances for all Workload Domains: vCenter Server NSX Manager SDDC Manager Optional: VMware Aria Suite components, Management Domain NSX Edge nodes The VI WLD vCenter and NSX Manager VMs are installed on the Management Domain hosts. Depending on the Workload Domain type, you can scale from 14 to 24 VI WLDs. A total of 1000 ESXi hosts are supported per VCF instance following the VCF standard architecture, where a single VI WLD can scale up to 800 ESXi hosts and up to 4000 registered VMs. A single VCF instance can have multiple availability zones (a stretched deployment).
Consolidated VCF SAP Domain
Represents a Management Domain that also runs customer workloads. It's limited to 100 hosts per consolidated VI WLD. Uses resource pools to ensure sufficient resources for management components. Considers the minimum possible initial hardware and management component footprint (min. 4 hosts). Only with vSAN and SAP HANA HCI certified hosts and the listed configurations for CPU, memory, and storage. Can be scaled to a standard architecture model. Management components and customer workloads are not isolated. SAP HANA production-level workloads must not run on the hosts that run the management components or non-SAP HANA VMs. This is an option for small deployments or test and dev environments. Add HANA dedicated hosts as required if you want to deploy production-level SAP HANA VMs.
SAP/SAP HANA VI Workload Domain
Represents an additional Workload Domain for running customer workloads. Can scale up to 14 VI Workload Domains per VCF instance. Shares a vCenter single sign-on domain and the identity provider configuration with the Management Domain. All Workload Domains can be managed through a single pane of glass. Can share an NSX Manager instance with other VI Workload Domains. A dedicated NSX Manager per VI WLD is recommended for easier lifecycle management. Allows for independent life cycle management of the different VI WLDs. This Workload Domain type cannot provide distinct vCenter single sign-on domains for customer workloads. For this, an isolated VI WLD is required. If SAP applications are deployed, only SAP-supported hosts can be used. SAP HANA hosts must also be SAP HANA TDI-supported systems and are limited to 2-, 4-, or 8-socket ESXi hosts with validated Intel CPUs. This Workload Domain type cannot provide distinct vCenter single sign-on domains for customer workloads. For this, an isolated VI WLD is required. If SAP applications are deployed, then only SAP-supported hosts can be used. SAP HANA hosts must also be SAP HANA TDI supported systems and are limited to 2-, 4- and 8-socket ESXi hosts with validated Intel CPUs. VM maximum host size: ≤ 12TB and 480 vCPUs per single SAP HANA scale-up VM. The smallest size is 0.5 socket with 128GB RAM. Scale-out deployment: VM must be ≥ 4 sockets, maximum 8 sockets, VM vRAM up to 6TB per VM depending on the use. Maximum 8 hosts + HA for a total size of 48TB. For shared ESXi or VM connected SAN or NAS storage w/o vVOLs, the storage must meet HANA KPIs. HCI certified hosts are required when vSAN is used. SAN or NAS deployments require min. 2 hosts; vSAN systems require min. 3 hosts. SAP NetWeaver and SAP HANA hosts can run consolidated in a VI WLD if the same SLAs and lifecycle management requirements exist. If not, then use dedicated VI WLDs for SAP HANA and NetWeaver or AnyApp.
SAP/SAP HANA Isolated VI WLD
Represents an additional Workload Domain for running customer workloads. Can scale up to 24 VI Workload Domains per VCF instance Has a distinct vCenter single sign-on domain and identity provider configuration. Workload domain vCenter Server instances are managed through different panes of glass. Can provide distinct vCenter single sign-on domains for customer workloads. Workload domains of this type cannot share an NSX Manager instance with other VI Workload Domains. Allows for independent life cycle management of the isolated VI WLDs and is the recommended option when a strict isolation of user and business data is required. If SAP applications are deployed, then only SAP-supported hosts can be used. SAP HANA hosts must also be SAP HANA TDI supported systems and are limited to 2-, 4-, and 8-socket ESXi hosts with validated Intel CPUs. VM maximum host size: ≤ 12TB and 480 vCPUs per single SAP HANA scale-up VM. Smallest size is 0.5 socket with 128GB RAM. Scale-out deployment: VM must be ≥ 4 sockets, maximum 8 sockets, VM vRAM up to 6TB per VM depending on the use. Maximum 8 hosts + HA for a total size of 48TB. For shared ESXi or VM-connected SAN or NAS storage w/o vVOLs, the storage must meet HANA KPIs. HCI certified hosts are required when vSAN is used. SAP NetWeaver and SAP HANA hosts can run consolidated in a VI WLD if the same SLAs and lifecycle management requitements exist. If not, then use dedicated VI WLDs for SAP HANA and NetWeaver or AnyApp.

Depending on the Workload Domain type, you can scale from 14 (ELM) to 24 (isolated) VI WLDs. Refer to Maximums and Configuration Limitations, VCF Deployment Options (for example, deployments in multiple availability zones), and vSphere Design for VMware Cloud Foundation.

In total, 1000 ESXi hosts are supported per VCF instance following the VCF standard architecture, where a single VI WLD can scale up to 800 ESXi hosts and up to 4000 registered VMs.

As shown in the figure and table above, the VI WLD vCenter and the NSX Manager VMs are installed in the VCF Management Domain among the vCenter and the NSX Manager VMs to manage the VCF Management Domain.

Review the Maximums and configuration limitations and VCF Deployment Options (for example, deployments in multiple availability zones for configuration details).

As previously explained, virtualized SAP HANA systems are currently supported on VMware vSphere with up to 448 vCPUs and 12 TB RAM per VM on Intel Cascade / Cooper Lake and with up to 480 vCPUs and 8 TB on Intel Sapphire Rapids systems; the vSphere 7.0 U2 and vSphere 8.0 U1 VM guest limits are 768 vCPUs and 24 TB per VM. Only the ESXi host systems shown in table 2 are validated on SAP HANA.
 

Note: The following may limit the maximum number of vCPUs and vRAM available for a VM: 

• The selected CPU type
• The virtualized SAP HANA workload type (OLTP or OLAP)
• An SAP HANA use case of network-heavy OLTP workloads with thousands of concurrent users, which may be required to reserve CPU threads to handle such an intensive network load.
• The implemented design options in VCF, like vSAN or NSX networking and security features, will reduce the available memory per SAP HANA VM.

Larger SAP HANA systems can leverage SAP HANA extension nodes or be deployed as SAP HANA scale-out configurations. In a scale-out configuration, up to 16 nodes (more upon SAP approval) work together to provide larger memory configurations. A scale-out SAP HANA node’s memory size depends on the selected CPU generation, and 4- or 8-socket systems with memory sizes per host and up to 6TB can be chosen. Refer to the relevant SAP Notes (3102813 and 3372365) for detailed information on supported host configurations. In addition, the SAP HANA on vSphere SAP Help Portal page provides an overview of supported configurations and is a good starting point.

An SAP HANA system deployed on a VMware SDDC based on VMware Cloud Foundation can be easily automated and operated by leveraging VMware Aria products. SAP HANA or hardware-specific management packs allow a top-to-bottom view of a virtualized SAP HANA environment, where an AI-based algorithm allows the operation of SAP HANA in a nearly automated approach, optimizing performance and availability. A tight integration with SAP Landscape Management Automation Manager via the VMware Adapter for SAP Landscape Management helps to cut down operation costs even further by automating work-intensive SAP management and operation tasks.

An SAP HANA system deployed in VMware Cloud Foundation allows you to adopt cloud-like virtual networking with NSX overlay segments, and it protects your business-critical SAP workloads with the NSX Distributed Firewall (DFW).

Note: This reference guide does not include the adoption of NSX vLAN, overlay segments, and NSX Distributed Firewall (DFW). These NSX deployment options require additional consideration and are currently out of the scope of this document.

In addition to SAP HANA, most SAP applications and databases can be virtualized and are fully supported for production workloads. Each SAP workload can run on its own ESXi host, or multiple SAP workloads/VMs can be consolidated on a single ESXi host.

Virtualizing all aspects of an SAP data center is the best way to plan for quick growth and easily move to a cloud-based infrastructure. SAP applications can also run in a true hybrid mode, where the most important SAP systems still run in the local data center and less critical systems run in the cloud.

SAP HANA on VCF deployment options and sizes

You can install SAP HANA on SAP-supported vSphere versions and validated CPU generations as scale-up and scale-out deployments on a single large VM or on multiple SAP HANA VMs on a single ESXi host. You may use only 2-, 4-, and 8-CPU socket VMware and SAP-supported or certified systems for SAP HANA production-level systems.

SAP HANA tenant databases (MDC) can run inside a VMware VM (see SAP Note 2104291, FAQ doc, page 2). Running SAP HANA VMs next to non-SAP HANA VMs, such as vSphere management VMs or SAP application servers, is also supported when these VMs run on different CPU sockets or when an SAP HANA and SAP NetWeaver AS (ABAP or Java) run in one VM (see SAP Notes 1953429 and 2043509).

The following table shows the supported host configurations for SAP HANA on vSphere and the deployment options for both single-tenant and multi-tenant SAP HANA instances on vSphere. It also gives guidelines for the standard memory sizes that SAP and VMware support based on the current SAP-defined memory limits for the top-bin Intel CPUs listed as certified appliance configurations.

The examples show the SAP-selected, top-bin CPUs with the maximum core count and memory support available for SAP HANA workloads. Refer to the SAP HANA Certified and Supported SAP HANA Hardware directory for details. Lower-bin CPUs or other CPU families may have different SAP HANA-supported memory configurations. As mentioned, deviating from these memory configurations is possible when an SAP HANA expert workload-based sizing is performed.

Note: The maximum available memory for a virtualized SAP HANA system is limited to the maximum memory tested with vSphere, which is currently 12TB per single scale-up VM or 48TB with scale-out deployments.
 

Legend:

Table 7. Overview of SAP HANA on vSphere deployment options and possible VM sizes

Note: Starting with the Intel Sapphire Rapids platform, SAP HANA "half-socket VM" support is only available with Intel sub-NUMA Clustering (SNC-2) enabled on vSphere 8 or newer ESXi 2-socket hosts.

SAP HANA half-socket VM support on non-SNC 2-socket Sapphire Rapids hosts or >2-socket Sapphire Rapids systems are not yet supported.

NUMA node/CPU sharing half-socket VMs are supported up to Ice Lake CPUs with vSphere 7 and, where applicable, with vSphere 8.
 

Intel sub-NUMA clustering (SNC-2) support and deployment options

Support for SNC-2

Support for SNC-2 in SAP HANA considers the increasing density of processors, memory controllers, processor interconnects, and supporting infrastructure within a single chip as the CPU size decreases. Increasing the number of CPU cores and, consequently, CPU performance is advantageous for users. However, this increased density is associated with a longer data transfer time between different parts of the CPU chip.

The process of accessing the memory within a CPU is facilitated by a uniform memory access (UMA) domain, which offers a unified and contiguous address space that is interleaved among all the memory controllers. UMA lacks a mechanism for optimizing data flow from the closest available resources. Processor affinity is typically employed to specify the processor(s) that a particular software thread utilizes. Consequently, in a Uniform Memory Access (UMA) system, all cores have equal access to the last level cache and memory. In this scenario, a processor might access a memory controller or a portion of the last level cache located on the opposite side of the CPU chip compared to the nearest.

Accessing and moving data within a CPU chip takes longer, and the memory subsystem is used more than before. This is clear when you compare the latest Sapphire Rapids CPUs to Cascade Lake CPUs. In this case, the number of CPU cores and memory controllers doubled (from 2 to 4 memory controllers) or slightly more than doubled (from 28 to 60 cores), but the number of memory channels only went up by 33.33%. The Sapphire Rapids platform has a faster memory bandwidth (MT/s), but this means that more CPU threads need to share a memory channel. DDR5 memory modules also have a longer CAS latency than DDR4 modules. CAS latency is a timing parameter that measures the delay between a memory controller sending a request for data and the memory module responding with the requested data. A longer CAS latency leads to bigger differences when memory-sensitive applications are used, for example, when two SAP HANA VMs share a NUMA node or CPU socket. A Sapphire Rapids CPU performs much better than a Cascade Lake CPU. This includes half-socket deployments.

However, there are more fluctuations when running several SAP HANA VMs on a socket. Because of these fluctuations, SAP does not support half-socket deployments for Sapphire Rapids CPUs. The following table shows this. It also shows a Sapphire Rapids CPU's theoretical available memory channels per CPU core.

The table doesn't reflect the change in increased memory speed. It only shows that the available memory channels are more utilized because of the increased CPU core count per chip and the higher CAS latencies when first accessing data stored in memory.

Table 8. Overview of CPU memory channels and CAS latency of selected SAP HANA-relevant CPUs

[1] For more information, refer to this article: The difference between RAM speed and CAS latency.

The number of memory channels refers to the bandwidth operation for the real-world application. (After following the hyperlink, click on (?) at Max # of Memory Channels.)

Intel introduced sub-NUMA clusters (SNC) to decrease the latency of data movements across the CPU. Sapphire Rapids CPUs support SNC-2 and SNC-4, but SAP HANA vSphere VMs support only SNC-2 on 2-socket host configurations.

Two localization domains exist within a processor in a two-cluster SNC (SNC-2). Each domain has addresses mapped from the local memory controller and local Last Level Cache (LLC) slices. Processors in the local domain will use the local memory controller and local LLC slices. This means that LLC and memory accesses in the local domain will have lower latency than accesses to locations outside the same SNC domain.

SNC has a unique location for every address in the LLC, and it is never duplicated within the LLC banks. Localization of addresses within the LLC for each SNC domain applies only to addresses mapped to the memory controllers in the same socket. All addresses mapped to memory on remote sockets are uniformly distributed across all LLC banks independent of the SNC mode. Therefore, even in the SNC mode, the entire LLC capacity on the socket is available to each core, and the LLC capacity reported through the CPUID is not affected by the SNC mode.

Refer to figure 2 – "Block Diagram Representing Domains Of sub-NUMA With Two Clusters" and figure 3 "– Block Diagram Representing Domains Of sub-NUMA With Four Clusters" on the Intel product webpage, which shows the block diagrams of a sub-NUMA node enabled CPU. Figure 2 represents a two-cluster configuration. This is the SAP-supported SAP HANA on vSphere configuration consisting of SNC Domains 0 and 1 and their associated cores, LLCs, and memory controllers. Each SNC-2 domain contains half of the processors on the socket, half of the LLC banks, and half of the memory controllers with its associated DDR channels, limiting a VM to these resources.

According to Intel, the affinity of cores, LLC, and memory within a domain are expressed via software using the NUMA affinity parameters in the operating system.

Note: SNC is enabled at the BIOS level and requires that the memory be symmetrically populated.

When SNC-2 is used for SAP vSphere VMs on 2-socket Sapphire Rapids, then we advise that either all hosts in a Sapphire Rapids vSphere cluster are SNC-2 enabled or that a VM host rule is applied to ensure that the SNC-2-enabled VMs do not get migrated or started on non-SNC-enabled hosts.

SNC-2 supported deployment examples

SAP HANA and SAP Application servers are supported by SAP and VMware with SNC-2 only on 2-socket Sapphire Rapids servers with vSphere 8 and can be leveraged as described in the table below.

The SAP HANA reference architecture for 2-socket Intel Xeon Platinum 8490H Processor Sapphire Rapids–based systems defines a 60-core Sapphire Rapids CPU to support up to 2TB of memory (4TB in total per 2-socket Sapphire Rapids host) according to SAP HANA appliance sizing. Different memory sizes and lower core count CPUs, like a 32- or 48-core Sapphire Rapids CPU, can also be used, but they require SAP application/workload sizing.

Note: The following configuration options for running SAP HANA on vSphere are not supported on bare metal servers enabled for SNC. Sapphire Rapids 4-socket hosts support half-socket deployments without SNC-2. Therefore, the following table isn't relevant for >2-socket Sapphire Rapids hosts. It only applies to 2-socket Sapphire Rapid hosts.

Legend:

 Table 9. SAP HANA–supported SNC-2 configurations

Important: Due to the lower memory bandwidth associated with an SNC-NUMA node, we recommend primarily using SNC-2 for half-socket VMs. If SAP HANA VMs running on SNC-2-enabled hosts experience performance issues, they should be migrated to a non-SNC-enabled host. All VMs that leverage SNC-2 require the VMX advanced parameter sched.nodeX.affinity="Y" to prevent unwanted NUMA node migrations.

If an SAP HANA VM running on an SNC-2-enabled host shows performance issues, migrate the VM to a whole socket non-SNC-enabled host as part of the troubleshooting.

Recommended: Build a vSphere cluster based on the same CPU model and generation. This allows for easy VM migration.

When you deploy SNC-2-enabled Sapphire Rapids systems, you must build a dedicated 2-socket SNC-2-enabled Sapphire Rapids cluster to ensure seamless VM migration. We don’t recommend adding non-SNC-enabled hosts to this cluster.

Review the "SNC-2 management and operation" section to learn how to best build and operate a vSphere SAP HANA cluster with SNC-2-enabled hosts and VMs, along with non-SNC-2 hosts and VMs.

Scale-up and scale-out deployment architecture

The following two figures describe typical scale-up and scale-out SAP HANA on vSphere architectures.

The storage must be SAP HANA certified. In the case of vSAN, the complete solution (server hardware and vSAN software) must be SAP HANA HCI certified.

The network column highlights the network needed for an SAP HANA environment. Bandwidth requirements should be defined regarding the SAP HANA size; for example, vMotioning a 2 TB SAP HANA VM or a 12 TB SAP HANA VM will take significantly more time to be migrated or will need a higher bandwidth network to minimize the migration time. Latency should be as low as possible to support the transactions of heavy or sensitive workloads and use cases. For SAP applications, the average response time for dialog transactions (online transactions) should be below one second.

The following figure shows scale-up SAP HANA systems (single SAP HANA VMs).

Figure 3. High-level architecture of a scale-up SAP HANA deployment on vSphere

The next figure shows a scale-out example where several SAP HANA VMs work together to build one large SAP HANA database instance. 

Figure 4. High-level architecture of a scale-out SAP HANA deployment on ESXi 4- or 8-socket host systems

Configuration and sizing guidelines

You must select the correct components and configuration to achieve the performance and reliability requirements for SAP HANA. The server configuration (RAM and CPU), network configuration, and storage configuration determine which and how many SAP HANA VMs can be supported.

Note: It is possible to consolidate a certain number of SAP HANA VMs on an ESXi host with a given RAM and CPU configuration; however, the network and storage configurations must have the capacity to support these SAP HANA VMs. Otherwise, a network or storage bottleneck could impact the performance of all running SAP HANA VMs on the host.

Sizing compute and memory

It is possible (since SAP HANA TDI Phase 5) to perform a workload-based sizing (SAP Note 2779240) and not depend on appliance configurations with fixed CPU-to-memory ratios.

You perform VMware virtual SAP HANA sizing in much the same way as you do for physically deployed SAP HANA systems. The significant difference is that an SAP HANA workload must fit into the compute and RAM maximums of a VM. You also need to consider the costs of virtualization (RAM and CPU of the ESXi host) when planning an SAP HANA deployment.

If an SAP HANA system exceeds the available resources for virtual deployments, the VM can be moved to a new host with more memory or higher-performing CPUs. After this migration to the new host, the VM must be shut down, and the VM configuration must be changed to reflect these changes (more vCPU and/or virtual memory). If a single host cannot satisfy the resource requirements of an SAP HANA VM, then you can use scale-out deployments or SAP HANA extension nodes.

Sizing process

As noted, sizing a virtual SAP HANA system is just like sizing a physical SAP HANA system plus the virtualization costs of CPU and RAM on the ESXi host). The following figure depicts the SAP HANA sizing process.

Figure 5. The SAP HANA sizing process

An SAP HANA sizing results in the following required resources:

• Compute - per the SAP Application Performance Standard (SAPS)
• Memory
• Storage

Note: By now, the SAP Quick Sizer allows you to calculate the needed SAPS for SAP HANA-based applications. Therefore, a manual calculation, as mentioned above in the table, is no longer required.

The SAP sizing tools do not cover network sizing. The SAP-provided network sizing information, which only focuses on throughput, is in the SAP HANA network requirements white paper. Network latency is only expressed as a general guideline—you must set your own goal. In SAP sales and distribution (SD) benchmarks, a time below 1,000 milliseconds for the average dialog response time must be maintained. See published SD benchmarks for the average response time.

In the Network configuration and sizing section, we refer to the SAP HANA network requirements white paper when we define the network infrastructure for a virtualized SAP HANA environment. SAP also provides a tool, ABAPMETER, to measure the network performance of a selected configuration to ensure it follows the SAP-defined and recommended parameters. See SAP Note 2879613: ABAPMETER in NetWeaver AS ABAP.

Note: The provided SAPS depend on the SAP workload you use. This workload can have an OLTP, OLAP, or mixed workload profile. From the VM configuration point of view, only the different memory, SAPS, network, and storage requirements are important and not the actual workload profile.

The storage capacity requirements for virtual or physical SAP HANA systems are identical. However, the CPU resource and physical memory requirements of a virtualized SAP HANA system are slightly higher than when deployed on bare metal servers; the virtualized requirements also include the virtual CPU and memory of the VM. We can express the CPU requirements with a fixed factor like 3-10%, but the memory needs are more challenging to determine.

The following table shows the estimated memory requirements of an ESXi host running SAP HANA workloads on different server configurations. Unfortunately, you cannot determine the memory cost of ESXi upfront. The physical server configuration influences them—for example, the number of CPUs, memory size, and installed cards like HBAs or NICs—as well as the vSphere features like NSX networking and security or vSAN in use.

Table 10. Estimated ESXi host RAM needs for SAP HANA VMs based on the number of sockets and memory installed on the ESXi host

Translating the SAP sizing to a VM configuration

Let’s use the following example to translate an SAP HANA sizing of a 1,400 GB SAP HANA database system that needs 80,000 SAPS on compute, according to the SAP Quick Sizer results.

The following formula calculates the available memory for SAP HANA VMs running on an ESXi host:

Total available memory for SAP HANA VMs = Total host memory - ESXi memory need

For this example, let’s assume that a 4-socket server with 6TB of total RAM and a 2-socket server with 2TB of RAM are available for selection. We use the default ESXi memory requirements:

• For the 4-socket ESXi host with a total of 6TB of RAM, that’s around 384GB of ESXi memory needed.
• For the 2-socket ESXI host with a total of 2TB of RAM, that’s around 128 of ESXi memory needed.

Virtual memory configuration example

SAP HANA example system memory sizing report result:

• 1400 GB RAM SAP HANA example system memory requirement

Available host server systems:

• 4-CPU socket 28-core Cooper Lake ESXi host with 6TB physical memory
• 2-CPU socket 60-core Sapphire Rapids ESXi host with 2TB physical memory

Virtual RAM calculation for the SAP HANA VM:

• 5760 GB (6144GB – 384GB RAM) / 4 CPU sockets = 1440GB per CPU socket > 1400GB sized HANA RAM → SAP HANA VM configuration needs 1 CPU socket
• 1920 GB (2048GB – 128GB) / 2 CPU sockets = 960GB per CPU socket < 1400GB sized HANA RAM → SAP HANA VM configuration needs 2 CPU sockets

Temporary VM configuration:

• In the 4-socket server case, the memory of one socket is sufficient for the sized SAP HANA system.
• In the 2-socket server case, both CPU sockets must be used because one socket has only 960GB available.
• In both cases, you should select all available logical CPUs per CPU socket, like in the Cooper Lake case 56 vCPUs (28 cores plus hyperthreading).

After you've configured the memory for the SAP HANA VM, you verify that the available SAPS capacity is enough for the work your planned workload.

SAP and its partners measure a server's SAPS capacity and CPU by running benchmarks and performance tests that SAP defines. One example is the SAP SD benchmark, which is used for all SAP NetWeaver and BW/4HANA applications. The test results of a public benchmark can be published and used in training on SAP sizing.

After you know the SAP application's SAPS resource needs, you can apply the sizing to a VM configuration.

The following figure, which is from a published SD benchmark, shows a way to estimate the available SAPS capacity of a virtual CPU. The SAPS capacity depends on the used physical CPU and is limited by the maximum available vCPUs per VM.

 Figure 6. Physical SAPS to virtual SAPS example conversion

First, look up the SAPS results of a benchmark of the CPU you want to use. Then, divide this result by the number of cores of the selected CPU. In the figure 6 example, the Cooper Lake (8380HL) CPU has 28 cores, and the Sapphire Rapids (8490H) CPU has 60 cores. Use the number of cores as the divisor. This provides the SAPS capacity of a hyperthreaded physical CPU core (you can use two CPU threads or two logical CPUs).

To estimate the virtual SAPS capacity of these two CPU threads, subtract the ESXi CPU resource needs, which are between 3%–10% for OLTP or OLAP workloads. In the above figure, to make the sizing easier, we use 10% for the virtualization costs for compute, which is subtracted from the previous result (two CPU threads running on one physical CPU core).

To define the SAPS capacity of a single vCPU running on a single CPU core, subtract the hyperthreading gain, which could be as little as 1% for very low-utilized servers or more than 30% for very high-loaded systems. For the sizing example shown in the above figure, we assume a 15% hyperthreading gain. Removing this 15% from the 2-vCPU result provides the SAPS capacity of a single vCPU (virtual SAPS 1 vCPU) that runs exclusively on a CPU core.

Recommended: Use hyperthreading (this is the default). Make sure that numa.vcpu.preferHT=TRUE (per VM setting) is set for the SAP HANA VM to ensure NUMA node locality. This is especially important for half-socket VM configurations and VM configurations that don’t span all NUMA nodes of a server.  

CPU calculation examples

The following examples show how to calculate the number of vCPUs needed to power the provided SAPS resource needs of the given SAP HANA workload.

VM CPU (SAPS) configuration example

Assumed/sized SAP HANA system:

• 1400GB RAM SAP HANA system memory
• 80,000 SAPS

Available host servers:

• 4-socket 28-core Cooper Lake (8380HL) host with 6TB physical memory, 89,000 pSAPS (356,000 SAPS in total), and 224 logical CPUs
• 2-socket 60-core Sapphire Rapids (8490H) host with 2TB physical memory, 212,000 pSAPS (424,000 SAPS in total), and 240 logical CPUs

VM vCPU configuration examples

4-socket 28-core Cooper Lake (8380HL) host:

• HANA CPU requirement as defined by sizing: 80,000 SAPS
• 356,000 / 4 = 89,000 SAPS / 28 cores = 3178 pSAPS per core
  vSAPS per 2 vCPUs (including HT) = 2860 SAPS (3,178 SAPS – 10%)
  vSAPS per vCPU (without HT) = 2,431 SAPS (2,860 SAPS – 15%)
• VM without HT: number of vCPUs = 80,000 SAPS / 2431 SAPS = 32,91 vCPUs, rounded up 33 cores / vCPUs
• VM with HT enabled: number of vCPUs = 80,000 SAPS / 2860 SAPS x 2 (threads per core) = 55,94 = 56 threads / vCPUs

VM vSocket calculation:

• 33 / 28 (CPU cores) = 1.18 or 56 (threads) / 56 (CPU threads) = 1
• The 1st result needs to get rounded up to 2 CPU sockets, since it does not leverage HT.
• The 2nd result uses HT and therefore the additional SAPS capacity of the hyperthreads is sufficient to use only one CPU socket for this system. This example shows why it is important to leverage hyperthreads.

2-socket 60-core Sapphire Rapids (8490H) host:

• HANA CPU requirement as defined by sizing: 80,000 SAPS
• 424k / 2 = 212k SAPS / 60 cores = 3533 SAPS core
  vSAPS per 2 vCPUs (including HT) = 3180 SAPS (3533 SAPS – 10%)
  vSAPS per vCPU (without HT) = 2700 SAPS (3180 SAPS – 15%)
• VM without HT: number of vCPUs = 80,000 SAPS / 2700 SAPS = 29,63 vCPUs, rounded up 30 cores / vCPUs
• VM with HT enabled: number of vCPUs = 80,000 SAPS / 3180 SAPS x 2 (threads per core) = 50,31 = 51 threads / vCPUs

VM vSocket calculation:

• 30 / 60 (CPU cores) = 0.5 or 51 (threads) / 120 (CPU threads) = 0.43

Considerations for the final VM configuration

The SAPS-based sizing exercise showed that without HT enabled and used by the VM, 2 CPU sockets of a Cooper Lake system are needed to fulfill the SAPS requirements. Only half of a 60-core CPU is needed for the Sapphire Rapids CPU. 

Recommended: Use hyperthreading in the VM configuration of the Cooper Lake system because, per the CPU requirements, only one CPU socket is sufficient.

The sized SAP HANA database memory of 1400GB is the leading HANA sizing and VM configuration factor. On the Cooper Lake system, we have 1440GB available per CPU. This fulfills the memory and CPU requirements as calculated. In contrast, the Sapphire Rapids system has only 2TB installed, which, per CPU socket minus the ESXi host, costs only 960GB. To fulfill the sized memory requirements of this example, you need to use 2 CPU sockets on the Sapphire Rapids host, which will result in poor performance. You could solve this problem by adding memory to this host.

The final VM configuration leverages all CPU and memory resources related to the CPU sockets needed. The resulting required configurations for the ESXi servers are:

• 4-socket 28-core Cooper Lake (8380HL) 6TB host: 1 CPU socket with 56 vCPUs (hyperthreading on) and 1440GB of virtual RAM
• 2-socket 60-core Sapphire Rapids (8490H) 2TB host: 2 CPU sockets with 240 vCPUs (hyperthreading on) and 1920GB of virtual RAM

Optimizing the Sapphire Rapids host configuration to have more memory per CPU socket will let you use a half-socket (SNC-2) configuration with a total of:

• 6TB per host and the possibility to run up to 4 VMs on this host, with each VM of around 1.5TB
• 6TB in the host with 2 VMs, each around 3TB, will result in low CPU performance (high CPU utilization). Therefore, if a half-socket SNC-2 deployment cannot be used, you can use a lower-bin Sapphire Rapids CPU.
• If you want to leverage SNC-2, then an SAP HANA half-socket VM with 1.5TB and enough SAPS performance would be the best case for this example SAP HANA VM.

Pre-sized examples

The following table shows possible VM sizes of specific CPU types and variants. We estimated the SAPS figures based on published SAP SD benchmarks and rounded them down. You'll need to subtract the RAM needed for the ESXi host from the shown figures. The figures represent the virtual SAPS capacity of a CPU core with and without hyperthreading.

Note: We based the SAPS figures shown in the following table on published SAP SD benchmarks; you can use these sizings for Suite on HANA, BW on HANA, S/4HANA, or BW/4HANA workloads.

• For half-socket SAP HANA configurations, you must subtract 15% from the SAPS capacity to consider the CPU cache misses caused by VMs running concurrently on the same NUMA node.
• For SNC-2 configurations that use half-socket VMs on Sapphire Rapids or other supported systems, this reduction is not necessary.
• For mixed HANA workloads, contact SAP or your hardware sizing partner.
• For SAP HANA on hyperconverged infrastructure (HCI), reserve an additional 10% SAPS capacity to take vSAN into account.

Use the following table to quickly determine a VM configuration that will fulfill the SAP HANA sizing requirements for RAM and CPU performance for SAP HANA on vSphere workloads.

Table 11. SAPS capacity and memory sizes for example configurations of SAP HANA on vSphere; based on published SD benchmarks and selected Intel CPUs

[1] Maximum NUMA nodes per CPU sockets depend on the Intel CPU architecture. Cascade Lake, Cooper Lake, and Sapphire Rapids CPUs support up to 8 socket CPU platforms. Ice Lake is a 2-socket platform only. Sapphire Rapids is only validated on up to 2-CPU sockets for SAP HANA on vSphere systems.

[2] SAP-supported appliance memory configurations are lower as specified by Intel. Cascade Lake and Cooper Lake CPUs support up to 1.5TB per CPU socket, Ice Lake up to 2TB, and Sapphire Rapids CPUs up to 3TB per CPU socket. 2TB are supported without a sizing; 3TB are supported in a mixed DIMM configuration with 128GB and 256GB DIMMs and require a workload-based SAP HANA sizing.

[3] The SAPS (SAP Application Performance Standard) figures provided are derived from published SD benchmark results, incorporating hyperthreading with a 2-vCPU configuration. To account for virtualization costs, a standard deduction of 10% is applied. For half-socket configurations, an additional deduction of 15% from the SD capacity is required, along with the standard 10% virtualization cost deduction. SNC-2 half-socket VMs are exclusive to 2-socket SPR systems and are calculated with a standard 10% deduction only. There's no need to subtract an additional 15%. All figures presented are rounded and based on rounded SAPS performance figures from published SAP SD benchmarks. These figures should be used exclusively for Suite on HANA, BW on HANA, or BW/4HANA workloads. For sizing parameters related to mixed HANA workloads, please consult SAP or your hardware vendor directly.

[4] The listed SAPS figures are based on published SD benchmark results with hyperthreading (2-vCPU configuration) minus a 10% virtualization cost. In the case of a half-socket configuration, in addition to the 10% virtualization cost, 15% from the SD capacity must be subtracted. SNC-2 half-socket VMs are calculated with only -10%.The shown figures are rounded and based on rounded SAPS performance figures from published SAP SD benchmarks, and can be used only for Suite on HANA, BW on HANA, or BW/4HANA workloads. For mixed HANA workload sizing parameters, contact SAP or your hardware vendor.

Here’s the VM configuration example again. This time, we use the information provided in the above table to quickly determine the VM configuration.

Sized SAP HANA system:

• 1400GB RAM SAP HANA system memory
• 80,000 SAPS (Suite on HANA SAPS)

Configuration example 1:

• 4-socket 28-core Cooper Lake (8380HL): 1.5TB per CPU host; total 6144GB
• 224 total CPU threads, 320,000 vSAPS available, 80,000 vSAPS per CPU by leveraging hyperthreading
• VM configuration: 1 socket; 56 vCPUs (80,000 vSAPS); 1400GB vRAM (optimal configuration)

Configuration example 2:

• 2-socket 60-core Sapphire Rapids (8490H) with 1TB per CPU host; total 2048GB
• 240 vCPUs; 381,000 vSAPS available; 95,000 vSAPS per half-socket by leveraging hyperthreading
• VM configuration: 2 sockets; 240 vCPUs; 1400GB vRAM (non-optimal configuration due to the memory configuration of the host). We recommend you upgrade the memory to 3TB per CPU.

Note: Review Broadcom KB 55767 for details on the performance impact of using vs. not using hyperthreading. As described, the performance impact of using hyperthreading is very little, especially for systems with low CPU utilization.

Recommended: In most cases, use hyperthreading for SAP HANA on vSphere hosts. You might make an exception for:

• Small workloads, which don’t require hyperthreading to increase performance.
• Workloads with very low latency requirements.
• Security risks exposed by hyperthreading that exist on some processors.

Storage configuration and sizing

Sizing a storage system for SAP HANA is different from sizing storage for classic SAP applications: SAP defines strict storage key performance indicators (KPIs) for production-level SAP HANA systems regarding data throughput and latency.

Important: Your production storage systems for SAP HANA VMs must meet these key performance indicators (KPIs). If multiple SAP HANA VMs are running on one host, the TDI storage must be capable of supporting the specified number of SAP HANA instances/VMs.

The only variable in storage sizing is the capacity, which depends on the size of the in-memory database.

Tools used to verify KPIs

You can verify your storage system meets the KPIs by using the following SAP tools:

• Hardware configuration check tool (HWCCT) for HANA 1.0
• Hardware and Cloud Measurement Tool (HCMT) for HANA 2.0

These tools are available only to SAP partners and customers. You need a valid SAP user account to download the tools, the KPI documentation, and the user guides. For details, see SAP Notes 1943937 and 2493172.

SAP partners provide SAP HANA-ready and certified storage solutions or certified SAP HANA HCI solutions based on vSAN  that meet the KPIs for a specified number of SAP HANA VMs. For more information, refer to the Certified and Supported SAP HANA Hardware web page.

Storage connection

Along with the storage capacity, you must plan the storage connection. Follow the available storage vendor documentation and planning guides to determine how many HBAs or NICs are needed to connect the planned storage. If no VMware-specific guidelines are available, use the guidelines for physically deployed SAP HANA systems as a basis, and work with the storage vendor on the final configuration supporting all possible SAP HANA VMs running on a vSphere cluster.

If, for example, you require a seamless migration between physically and virtually deployed SAP HANA systems, you can use vSphere-connected storage with the raw device, in-guest mapped LUNs, and in-guest mounted NFS storage solutions.

However, a fully virtualized storage solution works just as well as a natively connected storage solution and provides all the benefits of vSphere virtualization, including the possibility to abstract the storage layer from the operating system on which SAP HANA runs. For a detailed description of vSphere storage solutions, refer to the vSphere Storage documentation.

vSphere datastores

vSphere uses datastores to store virtual disks. Datastores provide an abstraction of the storage layer that hides the physical attributes of the storage devices from the VMs. For example, you can create datastores to be used as a single consolidated pool of storage, or you can use many datastores to isolate various application workloads.

vSphere datastores can be of different types: VMFS, NFS, vSAN, or vSphere Virtual Volumes. SAP HANA fully supports all of these. Refer to the Working with Datastores documentation for details.

The following table summarizes the vSphere features supported by the different storage types. All these storage types are available for virtual SAP HANA systems.

Note: For scale-out, you must install the SAP HANA shared file system on an NFS share. For all other SAP HANA scale-up and scale-out volumes, such as data or logs, you can use all storage types listed in the following table, as long as the SAP HANA TDI Storage KPIs are achieved per HANA VM. vSphere doesn't support other solutions, such as the Oracle Cluster File System (OCFS) or the IBM General Parallel File System (GPFS).

Table 12. vSphere-supported storage types and features

[1] Only vSAN-based SAP HANA-certified HCI solutions are supported. There is no support for generic vSAN solutions for SAP HANA production workloads.

In summary, you use storage with SAP HANA on vSphere to:

• Create separate and isolated datastores for operating systems, SAP HANA binaries, shared folders, data, and logs.
• Enable multiple SAP HANA VMs to provision their VMDK files on the same class of storage to meet SAP HANA storage KPIs.

File system layout

The following figure shows the recommended SAP HANA Linux file system layout when running SAP HANA on vSphere. We arranged the file system layout into three groups to help you decide when to use VMDK files or an NFS mount point to store the SAP HANA files and directories.

Figure 7.  SAP-recommended file system layout for SAP HANA on vSphere

Storage capacity calculation

All SAP HANA instances have a database log, data, root, local SAP, and shared SAP volume. The storage capacity sizing calculation of these volumes is based on the overall amount of memory needed by the SAP HANA in-memory database.

The storage subsystem configuration must meet very strict SAP-defined performance KPIs. This might result in more storage capacity than needed (even if the disk space is not needed, the number of disks might be required to provide the required I/O performance and latency).

SAP has published several architecture and sizing guidelines. The following figure and table consolidate that information—they show the typical disk layout of an SAP HANA system and the volumes needed. Use the information here as a starting point to plan the storage capacity needs for your SAP HANA on vSphere system. The volumes shown in the next figure should correspond with actual VMDK files and dedicated PVSCSI adapters or NVMe controllers, which ensures the best and most flexible configuration.

Recommended: Use NVMe controllers instead of PVSCSI because most modern systems have NVME devices instead of disks or SSDs installed.

The example configuration shown here uses three independent PVSCSI adapters or NVMe controllers and at least 4 independent VMDKs. This helps parallelize I/O streams by providing the highest operational flexibility. We also use the file system layout shown here and translate it into a disk volume/VMDK disk configuration.

Figure 8. Storage layout of an SAP HANA on vSphere system[1]

[1] Image sourced from SAP SE and modified.

The following table provides storage capacity examples of the different SAP HANA volumes. Some volumes, such as the operating system and usr/ SAP volumes, can be connected to and served by one PVSCSI adapter. Others, such as the log and data volumes, are served by dedicated PVSCSI controllers to ensure high I/O bandwidth and low latency.

Important: You must verify the performance of the I/O bandwidth and latency after an SAP HANA VM deployment with SAP hardware configuration checking tools. Refer to SAP Notes 1943937 and 2493172.

Instead of VMDK-based storage volumes, especially for data, you can use log or backup volumes that ingest connected NFS volumes.  
 

Table 13. Storage layout of an SAP HANA on vSphere system

[1] Information summarized from SAP OSS note 1900823, attachment: SAP HANA storage whitepaper version 2.1

Depending on the storage solution used and the future growth of the SAP HANA databases, you might need to increase the storage capacity or better balance the I/O over more LUNs. In this case, you can use the Linux Logical Volume Manager (LVM) to build LVM volumes based on VMDKs.

To determine the overall storage capacity per SAP HANA VM, sum the sizes of all specific and unique SAP HANA volumes as shown in the above figure and table.

To determine the minimum overall vSphere cluster datastore capacity required, multiply the SAP HANA volume requirements in the table by the number of VMs running on all hosts in the vSphere cluster.

Note: The raw storage capacity needed depends on the storage subsystem and the RAID level you used. Consult your storage provider to determine the optional physical storage configuration for running SAP HANA. NFS-mounted volumes do not need PVSCSI controllers.

Use the following equation for your storage capacity calculation:

vSphere datastore capacity = total SAP HANA VMs running in a vSphere cluster × individual VM capacity needed (OS + usr/sap + /shared + /data + /log)

For example, a sized SAP HANA system with RAM = 2TB would need the following:

• VMDK root (OS) >= 10GB (recommended 100GB, thin-provisioned)
• VMDK usr/sap >= 60GB (recommended 100GB, thin-provisioned)
• VMDK HANA shared = 1024GB (thick-provisioned)
• VMDK HANA log = 512GB (thick-provisioned)
• VMDK HANA data = 2048GB (thick-provisioned)
• VMDK HANA backup >= 2.5TB (thin-provisioned; optional)

In this example, the VM capacity requirement = 3.7TB with an optional backup for a total of 6.2TB.

To determine the cluster-wide storage capacity requirement, you must include all possible SAP HANA VMs running in the vSphere cluster in the calculation. All SAP HANA production VMs must fulfill the capacity, throughput, and latency requirements as specified by SAP Note 1943937.

Note: SAP HANA storage KPIs must be guaranteed for all production SAP HANA VMs. Use the HWCCT to verify these KPIs. Otherwise, the overall performance of the SAP HANA system might be lower than required. Refer to the previous section, "Tools used to verify KPIs."

Hardware and cloud measurement tool for SAP HANA

Using the SAP hardware check tools (refer to SAP Notes 1943937 and 2493172 and this blog post) allows you to verify that the storage performance and latency of an SAP HANA VM fulfills the SAP-defined KPIs for logs and data.

For its SAP HANA validation, VMware uses (besides vSAN) an external Fibre Channel flash array from Pure Storage. This ensures that the conducted tests and validations use a modern external storage array and software-defined storage.

The following table shows what is possible with a VMware-optimized Pure X50 storage system compared to a bare metal system running the same configuration and following the test results for log latency, which SAP has defined as a maximum of 1,000µs. SAP HANA running on a VM shows a 22% higher latency than SAP HANA running the same configuration on bare metal. However, the virtualized system still runs quite a bit below the SAP-defined KPI of 1,000µs, at 406µs, which shows good performance.

Table 14. HCMT file system latency example of SAP HANA running on a bare metal system vs on a vSphere VM

The next table shows that the virtualized SAP HANA system has only slightly higher throughput than the bare metal SAP HANA system running on the same configuration. However, both results are over the SAP-defined KPI of 120MBps for this test.

Table 15. HCMT file system overwrite example of an SAP HANA bare metal system compared to a virtualized system

For these tests, we used an 8-socket wide VM running on an Intel Cascade Lake Fujitsu PRIMEQUEST 3800B2 and lately 2- and 4-socket Intel Sapphire Rapids-based ThinkAgile VX650 and VX850 V3 systems from Lenovo connected via Fibre Channel on a Pure X50 storage unit configured as outlined in the best practices configuration guidelines. We conducted the tests for the results shown in the tables on an 8-socket Cascade Lake system. Make sure you apply the operating system best practice configuration for SLES or RHEL, and ensure you have a log and data volume per SAP HANA VM as previously described.

Network configuration and sizing

An SAP HANA VM on a vSphere cluster requires dedicated networks for the SAP application, user traffic, admin, and management, as well as for NFS or software-defined storage (SDS), such as vSAN, if it is used. Follow the SAP HANA network requirements technical paper to decide how many networks you need to support a specific SAP HANA workload in a VM and on the hosts.

Figure 9. Logical network connections per SAP HANA server[1]

[1] This figure was sourced from SAP SG.

In contrast to a physical SAP HANA environment, you must plan the SAP HANA operating system-exposed networks and the ESXi host network configuration that all SAP HANA VMs will share. You need to configure:

• The virtual network, especially the virtual switch, on the ESXi host
• HANA-exposed virtual network cards in the VM 

The following sections explain each.

Note: When configuring the host network, you should consider the need for multiple SAP HANA VMs on one ESXi host. A single SAP HANA instance on a single host might require more or higher bandwidth network cards.

vSphere offers standard and distributed switch configurations. You can use either when configuring an SAP HANA on vSphere environment.

Recommended: Use a vSphere Distributed Switch for all VMware kernel-related network traffic (such as vSAN and vMotion). A vSphere Distributed Switch acts as a single virtual switch across all associated hosts in the data cluster. This setup allows VMs to maintain a consistent network configuration as they migrate across multiple hosts.

VMs have network adapters that can be connected to port groups on the virtual switch. Every port group can use one or more physical NICs to handle their network traffic. Not having a physical NIC on a port group limits VM communication between the VMs and not the external network. You can find more information about this in the vSphere networking guide.

The next table describes the recommended network configuration for SAP HANA running virtualized on an ESXi host using different network card configurations. The information is based on SAP recommendations and includes vSphere-specific networks such as vMotion and dedicated storage networks for software-defined storage (SDS) or NFS. You might need to use multiple network cards to meet the network requirements for each SAP HANA VM when consolidating all of them onto a single ESXi host.

Note: This reference guide doesn't include the adoption of NSX networking (VLAN or overlay segments) or NSX Distributed Firewall (DFW). These NSX deployment options require additional consideration and planning and are currently out of scope for this document.

Table 16. Recommended SAP HANA on vSphere network configuration based on 10GbE NICs (non-VCF setup)

[1] The network card bandwidth influences how many SAP HANA VMs are supported on the vSAN datastore and how long a vMotion migration process will take to finish. Depending on the SAP HANA memory sizes, we recommend using 4- and 8-socket host systems and 4-socket large VMs with a minimum of 25GbE NICs.

Recommendations:

• Use VLANs to reduce the total number of physical network cards needed in a server.
• Make sure there’s enough network bandwidth available for each SAP HANA VM.
• Make sure the installed network cards of an ESXi host provide enough bandwidth to serve all SAP HANA VMs running on it. Oversubscribing network cards will result in poor response times or increased vMotion or backup times.

Note: The sum of all SAP HANA VMs that run on a single ESXi host must not oversubscribe the available network card capacity.

Recommended: Create a vSphere Distributed Switch per dual-port physical NIC and configure port groups for teaming and failover purposes. Use the default port group setting, except for the uplink failover order, as shown in table 17.

A port group defines properties regarding security, traffic shaping, and NIC teaming.

The following table shows an example of how to group the network port failover teams. It also shows the distributed switch port groups created for different functions and the respective active and standby uplinks to balance traffic across the available ones.

The suggestions in this table will differ depending on the required and optional networks needed for vSphere, SAP HANA system replication, or scale-out internode networks. At a minimum, 3 NICs are required for a virtual HANA system leveraging vMotion and vSphere HA. Additional network adapters are required for the optional use cases.

You should configure the network port failover teams to allow physical NIC failover.

Recommended: For physical NIC failover, you should use NICs with the same network bandwidth, such as only 10GbE NICs and group failover pairs, depending on their needed network bandwidth.

Important: Do not group 2 high-bandwidth networks, such as the internode network with the app server network.

Table 17. Minimum ESXi server uplink failover network configuration based on 10GbE NICs

[1] For the VLAN ID example, the final VLAN numbers are up to the network administrator.

Recommended: Use different VLANs, as shown, to separate the VMware operational traffic (for example, vMotion and vSAN) from the SAP and user-specific network traffic. Also, use higher bandwidth network adapters to reduce the number of physical network cards, cables, and switch ports. 

The next table shows an example with 25GbE network adapters.

Table 18. Example SAP HANA on vSphere network configuration based on 25GbE NICs

[1]The selected network card bandwidth influences how many SAP HANA VMs are supported on the vSAN datastore or how long a vMotion migration process will take to complete. Depending on the HANA memory sizes, we recommend you use 4- and 8-socket host systems and a 4-socket large VM with a minimum of 25 GbE NICs.

Using 25GbE network adapters helps reduce the number of NICs, network cables, and switch ports. With higher bandwidth NICs, the vSphere system can support more SAP HANA VMs per host.

The following table shows an example of how to group the network port failover teams.

Table 19. Minimum ESXi server uplink failover network configuration based on 25GbE NICs (VCF setup and non-VCF setup)

[1] The network administrator usually determines the final numbers for the VLAN ID example.

Recommended: Use different VLANs, as shown, to separate the vSphere operational traffic (for example, vMotion and vSAN) from the SAP and user-specific network traffic. Use 100 GbE bandwidth network adapters to further reduce the number of physical network cards, cables, and switch ports. 
 

The next table shows an example with 100GbE network adapters.

Table 20. Example SAP HANA on vSphere Network Configuration based on 100GbE NICs (VCF setup and non-VCF setup)

Note: The network example configurations in the tables above only leverage vSphere networking. For these examples, the configured host tunnel endpoint (TEP) for NSX overlay networking via VCF is irrelevant and not shown in the tables.
 

The following table shows an example of how to group the network port failover teams based on 100GbE NICs.

Table 21. Minimum ESXi Server Uplink Failover Network Configuration based on 100GbE NICs

[1] For the VLAN ID example, the final VLAN numbers are up to the network administrator.

Recommended: Use different VLANs, as shown, to separate the VMware operational traffic (for example, vMotion and vSAN) from the SAP and user-specific network traffic.

Workload performance and validation testing

The SAP HANA on vSphere validation involves different tests, some of which focus on CPU and memory performance, while others involve storage and network performance and scalability tests. These tests use SAP-specific OLTP and OLAP workloads and scale from single-user tests up to thousands of concurrent users (for example, up to 78,000 with Cooper Lake processors), pushing the virtualized SAP HANA system to its limits.

VMXNET3 network latency considerations: Our performance testing found that the virtualized network card (VMXNET3) typically adds between 60µs (no load) and up to 300µs of latency (high CPU load >= 80%) to every network package compared to a bare metal installed SAP HANA system, which impacts SAP OLTP and OLAP workloads issued by remote application servers or users. The following sub-sections include information about this impact and describe how to mitigate it by optimizing the virtual and physical network configuration.

SAP workload characterization and the impact on network performance

We can characterize SAP workloads in three ways:

• OLTP workloads, which represent classic SAP ERP systems
• OLAP workloads, which represent the analytical workloads of business warehouse (BW) systems
• Both OLTP and OLAP, which represent SAP S/4HANA systems

Because SAP S/4HANA combines these two workload types, we need to consider both of their characteristics. Typical OLTP workloads are small network packages with a recommended MTU size of 1,500, whereas the recommended MTU size for OLAP workloads is 9,000. You need to understand how the SAP S/4HANA system is used to choose the correct MTU size.

You also need to understand how many concurrent users will interact with the SAP HANA database and how much network traffic they will create. In recent tests of vSphere with SAP HANA with OLTP workloads, SAP and VMware observed an increase in OLTP transactional request times, which showed an overhead of up to 100ms when compared to bare metal SAP HANA systems.

We saw this rise when we used the VMXNET3 virtual NIC. This is because virtual networking adds latency to every network package sent or received, ranging from 60µs (no load) to up to 300µs (high load, wide VM, measured with 23,000–64,000 users running on a 4-socket Cooper Lake processor host and 35,000–78,000 users running on an 8-socket Cooper Lake processor host). This is documented in Broadcom KB 83957.

Unlike for storage (see the "Storage configuration and sizing" section), SAP did not define SAP HANA-specific network KPIs for throughput and latency that an SAP HANA system must maintain, apart from the general recommendation to use a 10 GbE network for SAP applications and SAP HANA database servers. Therefore, it is difficult to define a specific network configuration, and specific tests are required to recommend a suitable network configuration for a virtualized SAP HANA environment.

The next section discusses how we measured the VMXNET3 impact and how to optimize the network configuration for an SAP HANA VM for its given use case.

Note: This reference guide doesn't include the adoption of NSX networking (VLAN or overlay segments) or NSX Distributed Firewall (DFW). These NSX deployment options require additional consideration and planning and are currently out of scope for this document.

Performance tests run with SAP S/4HANA and BW/4HANA: workload and validation testing

We ran performance tests to measure the impact of virtualization on SAP HANA, and ultimately on the users, to define the best possible configuration for mitigating the virtualization costs, such as increased network latencies.

Testbed configuration

A typical SAP S/4HANA environment is a three-tier landscape, with the application tier on one or more hosts separate from the database tier. Users access the application servers when they work within the stored data of the SAP HANA database.

We configured tests with SAP S/4HANA and BW/4HANA to run on the 3-tier environment (shown below) to simulate real customer systems.

Figure 10. SAP 3-tier architecture on vSphere

The table below shows our testbed configuration.

Table 22. Testbed configuration

Workload information

One of the tests used for the validation is meant to simulate a day-in-the-life workload using common SAP online/interactive business transactions, such as the following:

• VA01: Create sales order
• VL01N: Create delivery
• VA03: Display order
• VL02N: Post goods issue
• VA05: List open orders
• VF01: Create invoice

Test methodology

The tests simulated OLTP and OLAP transactions up to the maximum possible CPU utilization level. A load driver initiated the application workloads, which simulated thousands of SAP users accessing the SAP HANA VM at the same time.

The application server instances received these requests and ran the SAP-specific transactions against the SAP HANA database. These transactions created several hundred database logical units of work (LUW) and were measured as a database request time in milliseconds.

Note: The measured database request time is the time taken for transactions between the SAP application server and the SAP HANA database instance. We did not measure the time a user needed to wait until an application server responded to a user-initiated transaction. The user-to-application server time is normally significantly higher than the database request time between application servers and a database instance.

The number of simulated SAP users per test run depended on the SAP HANA VM size and started at approximately 23,000 concurrent users for a 4-socket wide VM and approximately 35,000 concurrent users for an 8-socket wide VM (low load), and then increased to approximately 44,000 (4-socket) and approximately 60,000 (8-socket) concurrent users (moderate load), and then increased again to approximately 64,000 (4-socket) and approximately 78,000 (8-socket) concurrent users until the throughput dropped, which represents the maximum number of users possible (high load).

The number of concurrent users on the SAP HANA database instance represents a moderate to high CPU load:

• 35% CPU utilization → low CPU load
• 65% CPU utilization → moderate CPU load
• >65% CPU utilization → high CPU load (for example, 85%)

The number of users increased until the OLTP/OLAP throughout dropped. With this requirement:

• The 4-socket server achieved approximately 64,000 concurrent users at a CPU utilization of approximately 80%.
• The 8-socket system achieved approximately 78,000 concurrent users, also at a CPU utilization of about 80%.

As mentioned, these transactions were run on an 8-socket-wide Cascade Lake 8280L CPU-based server/VM until the maximum OLTP transactions per hour result was reached. The test-relevant results are of users at the 35% and 65% CPU utilization points. 

The result of the maxed-out measurement defines the maximum throughput number for OLTP/OLAP and the maximum possible number of SAPS for a physical or virtualized SAP HANA system.

Test results for OLTP workload

The test results of running the custom mixed workload and the BWH benchmark provide information on virtual network performance, allowing us to make recommendations to reduce the virtualization impact on networking performance.

Running the BWH benchmark didn't reveal any network throughput or latency issues, but the SAP S/4HANA custom mixed workload test revealed a network latency issue with the VMXNET3 driver, as documented in Broadcom KB 83957.

The following tables and figures summarize the results of these findings based on the latest Cooper Lake processors.

The following table shows the minimum and maximum VMXNET3 latency deviation measured with netperf on 8-socket-wide virtualized (VMXNET3 VM) and non-virtualized SAP HANA systems when running first with no load and next under a high user load. The high user load represents a CPU utilization of up to 65%. The bare metal SAP HANA system had, on average, a netperf TCP roundtrip time of around 26µs (no load) and up to 95µs (high load). On average, the virtualized SAP HANA system showed 84µs and 319µs TCP roundtrip time while under load.

Table 23. Network latency of Netperf TCP VMXNET3 compared to bare metal, measured in µs

The virtualization overhead here is 3x that of bare metal. The overhead specifically refers to the higher TCP round trip time (latency) per network package sent and received (while idle: no load; running: high load). This reduces the overall database request time performance. We found two ways to increase performance:

• Use a network card configured as passthrough. This shows only a slight latency compared to a physical NIC.
• Optimize the underlying physical network to lower the overall latency of the network. 

When we observed the VMXNET3 virtualization overhead (latency) of an 8-socket-wide VM running on an Intel Cooper Lake server with 416 vCPUs (32 CPU threads were reserved to handle this massive network load) to a natively installed SAP HANA system running on the same server, we saw how these microseconds accumulate on a database request time deviation of between 27ms and approximately 82ms. See the next figure for this comparison.

Figure 11. Mixed workload OLTP database request time in milliseconds (ms)

Using VMXNET3 resulted in a 25% (27ms) increase in the database request time at 35% CPU utilization but had no effect on the OLTP throughput per hour (TPH) or OLAP queries per hour (QPH). At approximately 65% CPU utilization, the database request time increased to 36% (82 ms higher) with a TPH deviation of approximately -1%. At the maximum user capacity, with 91,000 users and approximately 80% CPU utilization, the impact of TPH and QPH was approximately -8%. There was minimal impact on the OLAP request times. The following 3 figures show these results.

We found that using a network device configured as passthrough (instead of using a VMXNET3 network card) reduced the database request time for the OLTP application workload:

• At 35% CPU utilization, the VM with a passthrough NIC showed only a 3% difference (3 ms) compared to bare metal.
• At 65% CPU utilization, the VM with a passthrough NIC showed approximately a 9% (21 ms) difference.

80% of the time, the difference was still below 10%, keeping the THP and QPH deviations below -3% while maxed out.

Reserving CPU threads to handle network traffic on the ESXi side was not necessary because the network traffic was handled inside the VM and by the OS—not by ESXi as when VMXNET3 is used. Refer to the following figures.
 

The figure below shows that the VMXNET3 or passthrough network cards did not significantly impact the OLAP request times compared to a bare metal server.

Figure 12. Mixed Workload OLAP database request time in milliseconds (ms)

While the virtualization overhead can already be measured with lower user numbers (network traffic), this SAP S/4HANA mixed workload test has shown that the main impact can be measured at higher user load numbers that quickly generate massive OLTP requests and have little impact on OLAP database request time (next figure).

Figure 13. Mixed workload OLTP database TPH

Using a virtualized VMXNET3 network adapter typically has no significant effect on database request time at CPU utilizations between 35% and 65%, which means that the impact on OLTP throughput is less than 8%. In this test, the so-called max-out point was -8% using VMXNET3. Keep in mind that standard SAP workloads can reach 65% CPU utilization.

If your application is sensitive to database request times and you want to lower the network latency between the SAP app server tier and the database instance, consider using passthrough NICs instead of VMXNET3 NICs. However, passthrough doesn’t support vMotion.

The following figure displays the mixed workload OLAP QPH results of a Cooper Lake system. Using VMXNET3 has very little effect on the QPH results, which are between 35% and 65% (up to 1%), and at the max-out point at over 80% CPU utilization, it has about -8% of an impact.

Figure 14. Mixed workload OLAP database QPH

In summary, for OLAP workloads, a passthrough NIC doesn't make complex query runtime any faster. Reserving some CPU cores for the ESXi kernel by lowering the number of vCPUs in a VM could potentially improve performance during data loading, but it will result in lower QPH for SAP HANA.

For OLAP workloads, we recommend using the VMXNET3 NIC. OLTP workloads benefit most from passthrough NICs. If you don't use a passthrough NIC and need to use VMXNET3 for some reason, you could instead reduce the number of vCPUs.

We recommend you begin with VMXNET3, and if the database request time is longer than expected, then check the physical network infrastructure and, if possible, optimize it for low latency before considering passthrough NICs. This can help achieve nearly bare-metal NIC latencies. Once again, optimizing the SAP HANA database network for low latency and throughput will have the most beneficial impact on overall SAP HANA performance.

Note: When database request times are high, consider optimizing the physical SAP network. Start with the user, then move on to the SAP app server tier, and then from the app server to the SAP HANA database. Low-latency switches, a flat network architecture, or newer NICs will help to reduce the transaction request time experienced by users. The use of passthrough NICs inside the SAP HANA VM will only impact the database request time to the app servers, at the expense of losing vMotion capabilities. Additionally, using different CPU generations may yield different results. Sapphire Rapids systems show lower database request times (latencies) than older CPU generations.

SAP S/4HANA and B/4HANA validation test findings (OLAP)

Business warehouse (OLAP) workloads affect database request time and QPH results less than OLTP workloads—OLTP workloads generate more frequent network traffic and shorter packages. VMXNET3 latency overhead is less noticeable for OLAP workloads, which generate long-running queries but are less frequent.

The next table shows the certified results of published BWH tests with Intel Cascade Lake server systems. The results show a BWH configuration based on the L-class sizing that SAP specified for servers with 8 sockets and 6TB of memory.

We compared the virtual test results to those of a BWH system natively installed on the same hardware configuration.

The table below shows the test results. We ran the test on an Intel 8280L Cascade Lake server with 6TB of memory and 28-core CPUs running at 2.7GHz. The test consisted of 9 data sets (11.7 billion records).

Table 24. BWH 6TB L-class workload test: bare metal compared to VM with VMXNET3

As you can see in the above table, the virtualization overhead while using a VMXNET3 NIC was very low and within 10%. In Phase 3 (measured in seconds), there was no change in the total runtime of the complex query phase.

In very critical environments, this overhead can be lowered even further by leveraging a passthrough (instead of a VMXNET3) NIC.

BWH L-Class vs. M-Class sizings for SAP HANA VMs

The benchmark that HPE ran on SAP HANA 2.0 SP6 used 6TB and 12TB large database instances to store 10,400,000,000 (BWH L-Class sizing or 5,000 QPH) and 20,800,000,000 (BWH M-Class sizing or 2,500 QPH) initial records. This is the best result ever recorded in a vSphere 7 virtualized environment, with 7,600 (cert. 2022014) and 4,676 (cert. 2022015) queries per hour (QPH).

The following table and figure show the benchmark results of a bare metal vs. virtual environment. While the virtual results don’t pass the BWH L-Class sizing mark, they are still within 10% of a previously published bare metal BHW 12TB benchmark, which ran on the same hardware configuration with SAP HANA 2.0.

Table 25. BWH 12TB Cooper Lake, 8-socket M-Class workload test: bare metal vs. virtual

Figure 15. BWH 12TB Cooper Lake, 8-socket M-Class workload test: bare metal vs. virtual

With the new Sapphire Rapids–based systems with vSphere 8, we achieved the BWH L-Class sizing mark with both a 6TB and an 8TB 4-socket virtualized SAP HANA Sapphire Rapids system. With the 2-socket configuration, virtualized is very close to bare metal. See the next tables and figures for details. The overall virtualization overhead for the 2-socket configuration was below 3% of that for a bare metal configuration. For your reference, the 2-socket SPR 4TB BWH benchmark was published under cert number 2023030. 

Table 26. BWH 4 TB Sapphire Rapids, 2-socket M-Class workload test: bare metal vs. virtual

Figure 16. BWH 4TB Sapphire Rapids, 2-socket M-Class workload test: bare metal vs. virtual

The same benchmark running with more datasets on a 4-socket Sapphire Rapids system with 6TB and 8TB and 480 logical CPUs also shows very little virtualization overhead (below 6%) and with both memory configurations, we achieved the BHW L-Class CPU sizing category.

 
Table 27. BWH 8TB Sapphire Rapids, 4-socket M-Class workload test: bare metal vs. virtual, achieving L-Class CPU sizing mark

Figure 17. BWH 8TB Sapphire Rapids, 4-socket M-Class workload test: bare metal vs. virtual

Virtual SAP HANA performance evolution over 5 CPU generations

In summary, for our validation test results, we compared the different CPU generations to the same vSphere version using the SAP internal mixed workload tests and the public BHW benchmarks, which demonstrate significant performance gains.

The next tables provide a performance snapshot comparing different virtual SAP HANA systems running on Broadwell and Sapphire Rapids systems.

Table 28. 4-Socket vSphere 8 VM vs older CPU generations running mixed SAP HANA workload test VM @ max-out CPU MP

The following figure shows the throughput per hour (TPH) performance in this specific test for SAP HANA over the different CPU generations in graphical form. If you compare a 4-socket Intel Broadwell-based 4-socket server with an Intel Sapphire Rapids 4-socket system, then you see an increase in the TPH test result by a factor of 3, or 300%.

Figure 18. Performance evolution of CPU generations running mixed SAP HANA workload test VM @ max-out CPU MP measured in throughput per hour (TPH)

The next figure shows the queries per hour (QPH) performance gain over the different CPU platforms.

Figure 19. Performance evolution of CPU generations running mixed SAP HANA workload test VM @ max-out CPU MP measured in queries per hour (QPH)

The next table and figures provide the same overview but use the publicly available SAP HANA BWH benchmark. Apart from the storage used, the benchmark environment remained identical, making the most of the test results. Phase 1 of the BHW benchmark is comparable. Also, here we can see a significant improvement from an 8-socket Cascade Lake (448 logical CPU) system to a 4-socket Sapphire Rapids (480 logical CPU) system. Both systems run the same HANA, OS, and BHW versions with 6TB memory and 8 datasets configured. While the Sapphire Rapids system has more than 7.14% CPU threads, it shows a performance improvement of 35%. See the following figure and table for details.

Figure 20. BWH 6TB Sapphire Rapids, 4-Socket vSphere 8 VM vs. Cascade Lake, 8-Socket vSphere 8 VM - 35% performance gain

Table 29. BWH 6TB Sapphire Rapids, 4-socket vSphere 8 VM vs. Cascade Lake, 8-socket vSphere 8 VM

Using the latest Sapphire Rapids hosts with vSphere 8.0 U1 or later results in significant performance gains compared to previous CPU generations running the same SAP HANA workload. This not only allows for faster processing, but also larger memory configurations and a smooth transition from older CPU generations to Sapphire Rapids hosts. The performance boost from a Sapphire Rapids CPU mostly offsets the virtualization costs. So, you can migrate a bare metal SAP HANA installation to a virtualized Sapphire Rapids host, which saves you money because it requires less data center space.

When transitioning from an 8-socket Cooper Lake or Cascade Lake system, which can accommodate up to 12TB of memory, to a 4-socket Sapphire Rapids system, it's essential to note that the standard memory support for the latter is limited to 8TB.

However, with the assistance of the Technology Development Interface (TDI), a 4-socket Sapphire Rapids host can potentially handle higher memory capacities. SAP allows up to 3TB per CPU, indicating that a 4-socket Sapphire Rapids configuration could support up to 12TB of memory. Moreover, given the enhanced performance capabilities of the CPU, you could seamlessly perform such an upgrade.

Enhanced vMotion Compatibility, vSphere vMotion, and vSphere DRS best practices

A benefit of virtualization is the hardware abstraction and independence of a VM from its underlying hardware. Enhanced vMotion Compatibility, vSphere vMotion, and vSphere Distributed Resource Scheduler (DRS) are key enabling technologies for creating a dynamic, automated, self-optimizing data center. This allows a consistent operation and migration of applications running in a VM between different server systems without the need to change the operating system and application, or to perform a lengthy backup. A change in hardware would also add time because you'd need to update the device drivers in the operating system too.

vSphere vMotion live migration (shown in the following figure) allows you to move an entire running VM from one physical server to another with zero downtime, continuous service availability, and complete transaction integrity. The VM retains its network identity and connections, ensuring a seamless migration process. A high-speed network can transfer the VM's active memory and precise running state, allowing the VM to transition from running on the source ESXi host to the destination ESXi host.

Figure 21. vSphere vMotion allows the live migration of SAP HANA VMs

With enhanced vMotion compatibility mode, you can migrate SAP HANA VMs between ESXi hosts with different CPU generations. This lets you combine older and newer server hardware generations into a single cluster, thereby saving money because you don't need to replace the older ESXi hosts.

You can move a VM from one ESXi host to another in different modes. vSphere supports some modes for a powered-on SAP HANA VM. Other modes, such as migration to another storage system, have limited support while the VM is powered on. The next section provides an overview of the different VM migration options and what to consider when migrating SAP HANA VMs.

Best practices for migrating SAP HANA VMs in a production environment

Using vMotion, it's possible to migrate a VM between different hardware generations of a CPU type or storage subsystems.

• Run SAP HANA VMs within the vSphere cluster on identical hardware (with the same CPU clock speed and synchronized TSC). This ensures that SAP HANA has the same CPU features and clock speed available.
• Don't live migrate SAP HANA VMs while a virus scanner or a backup job is running inside the VM or while people are using the SAP HANA VMs, because this can cause a soft lock of the SAP HANA application.
• Don't live migrate SAP HANA VMs while a VM snapshot-based backup job is running outside the VM. This can cause a soft lock or data inconsistencies in the SAP HANA application.
• Use vMotion only during non-peak times (for example, when CPU utilization is less than 25%).
• If the source host and destination host clock speeds are different, you can still use vMotion. However, you should plan to upgrade the VM's hardware version and align the vCPUs to the new CPU so that you can restart the VM and use any new CPU features of the target host.
• You may use vSphere Storage vMotion to migrate SAP HANA VMs between storage subsystems. Because vSphere Storage vMotion impacts the performance of a running VM, we strongly recommend you perform a storage migration while the VM is powered off, or at least while the SAP HANA database is shut down inside the VM.
• Allocate sufficient bandwidth to the vMotion network, ideally 25 GbE or more.
• Avoid having "noisy neighbors" active on the ESXi host during a vMotion migration of SAP HANA VMs. A noisy neighbor is another VM that is using up the host's resources, leaving few left for other activities.
• Check SAP HANA patch levels (some patch levels may increase the risk of soft lockups to the operating system during migrations).
• Upgrade to vSphere 7 or 8 to leverage vMotion improvements.

VM migration scenarios

You can run a vMotion migration manually, fully, or semi-automated with vSphere DRS. To mitigate any potential impact on SAP HANA VMs during a VM migration, we recommend the following:

• Use vMotion only during non-peak times (low CPU utilization: less than 25%).
• Have vSphere DRS rules in place that:
  • Only suggest initial placement or
  • Allow VMs to be automatically moved when a host is set to maintenance mode.
• A dedicated vMotion network is a strict requirement. The network should have enough bandwidth to support a fast migration time, which depends on the active SAP HANA memory; for example, >= 4GB SAP HANA VMs. A vMotion network with 25GbE or higher bandwidth is preferred. Multiple vMotion network cards will help parallelize the vMotion process and lower the impact on the VM performance and time.

The following figures depict all the supported scenarios. We discuss what you should do, per scenario, to avoid possible performance issues.

Figure 22. Live migration of an SAP HANA VM (default scenario)  

Description:

• Typical VM migration scenario mainly used for load balancing or to transfer VMs off the host for server maintenance.
• All hosts have the same vSphere version and identical HW configuration.
• Hosts are connected to a shared VMFS datastore and are in the same network.

Considerations:

• Perform a manual vMotion migration during non-peak hours.
• Enhanced vMotion Compatibility is not required because all hosts use the same CPU.

Figure 23. Transferring VMs off the ESXi host to a new host with a new ESXi version 

Description:

• VM migration scenario to transfer VMs off a ESXi host (set to maintenance mode) before a vSphere upgrade.
• All hosts have an identical HW configuration.
• Hosts are connected to a shared VMFS datastore and are in the same network.

Considerations:

• Perform a manual vMotion migration during non-peak hours.
• Enhanced vMotion Compatibility is not required because all hosts use the same CPU.
• Live migration is possible, but double-check the virtual hardware version first:
  • Before you upgrade the virtual hardware, you might need to upgrade VMware Tools first. Double-check the dependencies in the release notes.
  • You might need to upgrade the virtual hardware of the VM; for example, hardware version 16 to 21.
  • You might need to align the virtual hardware to the new vSphere maximums; for example, 6 TB VM now to 12 TB.
  • You must restart the VM after you change its virtual hardware.

Figure 24. Migrating a SAP HANA VM during a host hardware upgrade

Description:

• VM migration scenario to migrate a VM to a new host with a different CPU (CPU type or new CPU generation). This is a critical vMotion scenario (see below).
• Hosts are connected to a shared VMFS datastore and are in the same network.

Considerations:

• Perform a manual vMotion migration during non-peak hours.
• EVC is required to allow the use of hosts with different CPUs in one vSphere cluster.
• Online migration of VMs is possible, but if the hardware has changed, then the VMs need to be aligned to the changed HW and CPU configuration.
  • Plan offline time for VM maintenance to perform a virtual hardware upgrade of the VM (for example, hardware version 20 to 21).
  • Align the number of vCPUs per socket and VM memory to the physical CPU capabilities and NUMA node memory configuration.
  • Power on the VMs on the new host.
• If the memory and CPU core configurations remained the same, then a simple restart of the VMs is enough to ensure the timestamp counter is synchronized; otherwise, the VM performance could be negatively impacted (more on this below).

If the hardware configuration of the host (CPU cores and/or memory or number of sockets) has changed, you need to power off the VM and change the VM configuration to align it to the new host configuration.

Background: By default, the TSC inside a VM runs at the same rate as the host when it powers on. If the VM is then moved to a different host without being powered off (for example, with vMotion), a rate transformation is performed so the virtual TSC continues to run at its original power-on rate, not at the new host TSC rate. For details, read the document Timekeeping in VMware Virtual Machines.

To resolve this issue, schedule a maintenance window to restart the VMs on non-identical hardware. This will enable the use of hardware-based TSC, instead of software rate transformation on the target host, which can be costly and lead to VM performance degradation.

Figure 25. The maintenance window as a hardware upgrade and vMotion

*Business downtime required

Figure 26. VMware VM migration - storage migration (VM datastore migration) 

Description:

• VM migration scenario to migrate a VM to a new host with a different VMFS datastore.
• Migration between local and shared or different shared storage subsystems possible.
• Hosts are connected to the same network.

Considerations:

• Requires manual vMotion during off-peak hours.
• EVC might be required if the hosts have different CPUs installed.
• Online migration is possible, but in the SAP HANA case, we don't recommend a migration with Storage vMotion.
• Plan offline time for VM maintenance to perform the storage migration. Storage vMotion takes more time and has a higher impact on VM performance.
• Ensure the new storage meets the SAP HANA TDI storage KPIs.

Managing SAP HANA landscapes using vSphere DRS

vSphere DRS, an automated load-balancing technology that aligns resource use with business priorities, can be used to manage SAP HANA landscapes. vSphere DRS dynamically aligns resources with business priorities, balances computing capacity, and reduces power consumption in the data center.

vSphere DRS takes advantage of vMotion to migrate VMs among a set of ESXi hosts. vSphere DRS continuously monitors utilization across ESXi hosts and can migrate VMs to less utilized hosts when they need more resources.

When deploying large SAP HANA databases or production-level SAP HANA VMs, it is essential to have vSphere DRS rules in place and to set the automation mode to manual. If you don't set the automation mode to manual and use the automated mode, then you must set the DRS migration threshold to conservative (level 1), to avoid unwanted migrations, which may negatively impact the performance of an SAP HANA system. It is possible to define which SAP HANA VMs should be excluded from automated DRS-initiated migrations and, if at all, which SAP HANA VMs are targets of automated DRS migrations.

DRS can be set to these automation modes:

• Manual: DRS recommends the initial placement of a VM within the cluster, and then recommends the migration. You must manually place and migrate the VM.
• Partially automated: DRS automatically places a VM when it is being powered on, but during load balancing, DRS displays a list of vMotion recommendations for you to select and apply.
• Fully automated: DRS placements and migrations are automatic.

SNC-2 management and operation

Building homogenous SAP HANA on vSphere clusters helps to streamline operations and eliminate possible issues when an SAP HANA VM is started on another host in the cluster. You must use the same CPU model and type, size of host memory, installed adapters like HBAs or NICs, and storage configuration. Adding SNC-2-enabled 2-socket Sapphire Rapids hosts to a cluster provides some benefits, like the co-deployment of other types of VMs (not SAP HANA VMs) on the same physical CPU socket. But it makes the operation of such an environment more challenging. To make your job easier when adding SNC-2 enabled hosts, use VM host rules. 

This VM-specific SNC-2 VM/host rule must be applied to all VMs that leverage SNC-2. This ensures the SNC-2-enabled VMs do not get migrated or started on non-SNC-enabled hosts. The next figure shows an example VM/host rule that ensures VMs created on SNC nodes run only on SNC-enabled hosts.

Figure 27. VM SNC-2 host rule

The following figure shows an example cluster with Sapphire Rapids-based 2- and 4-socket systems.

The 4-socket systems are used for the larger SAP HANA VMs that require more memory. All hosts adhere to the SAP HANA memory standard configuration of 2TB per CPU. OLAP workloads running on these systems, alongside the OLTP workload VMs, are verified with an SAP HANA OLAP expert sizing to ensure that 2TB per CPU can be utilized.

You should consolidate smaller SAP HANA VMs, in this case, up to 1TB, on the two SNC-2 hosts to benefit from optimized memory latencies. If an SNC-2 sub-NUMA node VM needs to be extended, you should migrate it to one of the 2- or 4-socket non-SNC-2 hosts, shown in the figure, to get the best possible improvements of a single NUMA node configuration with the same size. 

Figure 28. VMware SAP HANA vSphere 8 mixed Sapphire Rapids (SPR) cluster

Important: Due to the lower memory bandwidth associated with an SNC NUMA node, we recommend you primarily use SNC-2 for half-socket VMs. In the event of performance issues with SAP HANA VMs running on SNC-2-enabled hosts, you should migrate these VMs to a non-SNC-enabled host to resolve SNC-related performance issues. All VMs that leverage SNC-2 require the VMX advanced parameter sched.nodeX.affinity="Y" to prevent unwanted NUMA node migrations.

Guidelines for deploying an SAP HANA VM on an SNC-2 enabled 2-socket Sapphire Rapids ESXi host:

Note: SNC-2 is only required for 2-socket Sapphire Rapids hosts. >2-socket Sapphire Rapids hosts can deploy 2 SAP HANA VMs on a single NUMA node without SNC-2.

• SNC requires that the ESXi host memory be symmetrically populated.
• Enable SNC-2 and hyperthreading in the BIOS of the ESXi host.
• Use SNC-2 for SAP HANA VMs exclusively on 2-socket SPR hosts or later.
• Size the SAP HANA VM according to available logical threads and memory per CPU socket.
• Apply the sched.nodeX.affinity="Y" VMX advanced parameter to all VMs that leverage SNC-2 to prevent unintended NUMA node migrations.
• Use SNC-2 primarily as a consolidation platform for half-socket VMs or SAP HANA VMs requiring low memory latency due to lower memory bandwidth associated with an SNC NUMA node.
• SAP HANA VMs within a 2-socket SNC-2 host can be extended up to 4 sub-NUMA nodes (refer to figure 28, above, for details).
• Address SNC-2 performance issues by offline migrating SAP HANA VMs from SNC-2-enabled hosts to non-SNC-enabled hosts and adjusting the VM configurations accordingly to the non-SNC configuration.
• The vMotion of VMs that leverage SNC-2 is only supported between SNC-2 enabled hosts. Avoid migrating SNC-2-configured VMs to non-SNC-2 hosts to prevent performance issues. Migration in the reverse direction is also unsupported for the same reason.
• Use vMotion host rules to prevent the migration of SNC-2 VMs to non-SNC-2 ESXi hosts.

vSphere Cluster Services

Starting with vSphere 7.0 Update 1, vSphere Cluster Service (vCLS) is enabled by default and runs in all vSphere clusters.  vSphere makes critical cluster services, such as vSphere HA and vSphere DRS, highly available, and vCLS makes cluster services even more available.

SAP HANA, as the foundation of most SAP business applications, is a very critical asset of all companies using SAP solutions for their business. Due to the criticality of these applications, it is important to protect and optimally operate SAP HANA.

Running SAP HANA on vSphere provides an easy way to protect and operate it by leveraging vCLS, which depends on vCenter Server availability for configuration and operation.

The dependency of these cluster services on vCenter Server is not ideal, and the use of vCLS is the first step to decouple and distribute the control plane for clustering services in vSphere and to remove the vCenter Server dependency. If vCenter Server should ever become unavailable, vCLS will ensure that the cluster services remain available to maintain the resources and health of critical workloads.

Architecture of vCLS

vCLS uses agent VMs to maintain cluster service health. Up to three vCLS agent VMs, which are lightweight and build the cluster control plane, are created when you add hosts to clusters.

vCLS VMs must run and are distributed in each vSphere cluster. vCLS is also enabled on clusters that contain only one or two hosts. In these clusters, there are only two vCLS VMs. The next figure shows the high-level architecture with the new cluster control plane.

Figure 29. vCLS high-level architecture

A cluster enabled with vCLS can contain ESXi hosts of different versions if the ESXi versions are compatible with vCenter Server 7.0 Update 1. vCLS works with both vSphere Lifecycle Manager and vSphere Update Manager managed clusters and runs in all vSphere licensed clusters.

vCLS VM details

vCLS VMs run in every cluster, even if cluster services such as vSphere DRS or vSphere HA are not enabled on the cluster.

Each vCLS VM has 100MHz and 100MB capacity reserved in the cluster. For more details, refer to the Monitoring vSphere Cluster Services documentation.

In the normal use case, these VMs are not noticeable in terms of resource consumption. You are not expected to maintain the lifecycle or state for the agent VMs; they should not be treated like typical workload VMs.

Table 30. vCLS VM resource allocation

Table 31. Number of vCLS agent VMs in clusters

Guidelines for deploying vCLS in a cluster with SAP HANA VMs

As of SAP notes 3102813 and 3372365, you can't run a non-SAP HANA VM on the same NUMA node where an SAP HANA VM already runs: "SAP HANA VMs can be co-deployed with SAP non-production HANA or any other workload VMs on the same vSphere ESXi host, if the production SAP HANA VMs are not negatively impacted by the co-deployed VMs. In case of negative impact on SAP HANA, SAP may ask to remove any other workload." Also, "no NUMA node sharing between SAP HANA and non-HANA is allowed." 

Note: These statements do not apply to a 2-socket Sapphire Rapids platform with SNC-2. Here, it is possible to co-deploy a non-SAP HANA VM like an SAP application server or a VMware vCLS VM. Co-deployment of other VMs, like a web server, is still not supported.

All other CPU generations and >2-socket Sapphire Rapids systems have restrictions because of SAP guidelines and the mandatory and automated installation process of vCLS VMs. When upgrading to vCenter 7.0 Update 1, you must check if vCLS VMs were co-deployed on ESXi hosts that run SAP HANA production-level VMs. If this is the case, then you must migrate these VMs to hosts that do not run SAP HANA production-level VMs or to another SNC-2 CPU socket host when SNC-2 is used.

You'll need to configure vCLS VM anti-affinity policies. These policies describe a relationship between VMs that have been assigned a special anti-affinity tag (for example, a tag named "SAP HANA") and vCLS VMs.

When you assign this tag to SAP HANA VMs, the policy discourages placement of vCLS VMs and SAP HANA VMs on the same host. After you create the policy and assign tags, the placement engine will attempt to place vCLS VMs on the hosts where tagged VMs are not running—for example, the HA ESXi host. For this to work, you need to have some VMs on the cluster that are not tagged with "SAP HANA".
 

Examples of deploying vCLS in a cluster with SAP HANA VMs

Typically, you deploy SAP HANA VMs on dedicated ESXi hosts. These hosts can be part of small or large clusters/VCF VI WLD (in terms of number of hosts). They can be mixed with hosts running non-SAP HANA workload VMs or can be part of a dedicated SAP HANA cluster/VCF VI WLD.

The following examples of typical SAP HANA clusters provide some guidelines on where to place up to three lightweight vCLS VMs.

Mixed SAP HANA and other VMs in a vSphere cluster/VCF VI WLD

A mixed cluster/VCF VI WLD with SAP HANA VMs and other VMs is a typical scenario. In this case, check the vCLS VMs to see if they were deployed on ESXi hosts that run production-level SAP HANA VMs. If yes, then the vCLS VM can run on the same CPU socket as an SAP HANA VM.

To avoid this, configure vCLS anti-affinity policies:

1. Create a category and tag for each group of VMs that you want to include in a vCLS VM anti-affinity policy.
2. Tag the VMs that you want to include.
3. Create a vCLS VM anti-affinity policy.
   1. From vSphere, click Policies and Profiles > Compute Policies.
   2. Click Add to open the New Compute Policy wizard.
   3. Fill in the policy name and choose vCLS VM anti affinity from the Policy type drop-down control. The policy name must be unique.
   4. Provide a description of the policy, then use VM tag to choose the category and tag to which the policy applies. Unless you have multiple VM tags associated with a category, the wizard fills in the VM tag after you select the tag category.
   5. Click Create to create the policy.

The following figure shows the initially deployed vCLS VMs and how these VMs get automatically migrated (green arrows) when the anti-affinity rules are activated to comply with SAP notes 3102813 and 3372365. Not shown in this figure are the HA host/HA capacity reserved for HA failover situations.

Note: If you need to add new hosts to an existing SAP-only cluster/VCF VI WLD to make it a mixed host cluster, ensure that you  verify the prerequisites as outlined in the Add a Host to a Cluster documentation.

Figure 30. vCLS VM migration for a mixed host cluster: migrate the vCLS VMs to an ESXi host that runs workloads other than SAP HANA VMs

Dedicated ESXi hosts for SAP HANA VMs on a vSphere cluster

You might have deployed a vSphere cluster with ESXi hosts that run only SAP HANA VMs. In this scenario, the automatically deployed vCLS VMs reside on an SAP HANA host, making their migration to hosts without SAP HANA VMs challenging. The solution is to add existing hosts with non-SAP HANA workload VMs to this cluster, or to have non-tagged SAP HANA, non-production VMs running on at least one host. These existing hosts may run any workload, such as a SAP application server VM or infrastructure workload VMs. You don't need to buy a new host for this.

The next figure shows the initially deployed vCLS VMs and how these VMs are moved when the vCLS anti-affinity policy for SAP HANA VMs is added to new hosts. 

Figure 31. Migrating vCLS VMs in a dedicated SAP HANA host cluster

SAP HANA HCI on a vSphere cluster

Just as with the dedicated SAP HANA cluster, an SAP HANA HCI cluster may only run SAP HANA workload VMs in a VCF Workload Domain. As with SAP HANA running on traditional storage, SAP HANA HCI (SAP note 2718982) supports the co-deployment with non-SAP HANA VMs as outlined in SAP notes 2937606 (vSphere 7.0) and 2393917 (vSphere 6.5/6.7).

If vCenter is upgraded to 7.0 U1, then the vCLS VMs will be automatically deployed on SAP HANA HCI nodes. If you exclusively use these nodes for SAP HANA production-level VMs, you must remove the vCLS VMs and migrate them to the vSphere HCI HA host (or hosts).

You can do so by configuring vCLS VM Anti-Affinity Policies. A vCLS VM anti-affinity policy describes a relationship between VMs that have been assigned a special anti-affinity tag (for example, the tage name "SAP HANA") and vCLS system VMs.

If this tag is assigned to SAP HANA VMs, the vCLS VM anti-affinity policy discourages the placement of vCLS VMs and SAP HANA VMs on the same host. Such a policy prevents the co-deployment of vCLS VMs and SAP HANA VMs. After you create the policy and assign tags, the placement engine attempts to place vCLS VMs on hosts where tagged VMs are not running, such as the HCI vSphere HA host.

If you are unable to add an additional non-SAP HANA ESXi host to the cluster or need to validate a SAP HANA HCI partner system, you can use Retreat Mode to remove the vCLS VMs from this cluster.

In summary, by introducing vCLS, Broadcom is embarking on a journey to remove the vCenter dependency and possible related issues when vCenter Server is not available and provide a scalable platform for larger ESXi host deployments. When the ESXi host is part of a VCF VI WLD/cluster, the vCLS VMs cannot be set to retreat mode because the VCF lifecyle management depends on DRS to automatically put the host in maintenance mode.

For more information, please see the following resources:

• vSphere Clustering Services
• vSphere Clustering Service VM Anti-Affinity Policies
• Create or delete a vSphere Clustering Service VM Anti-Affinity Policy

High availability best practices for virtualized SAP HANA

SAP HANA offers several methods for high availability and disaster recovery: auto-failover, service restart options, backups, system replication, and standby host systems. In VMware virtualized environments, you can use any of these solutions. In addition, you can use vSphere HA and vSphere Replication to minimize unplanned downtime due to faults.

There are two areas of high availability support: fault recovery and disaster recovery.

Fault recovery includes:

• SAP HANA service auto-restart
• Host auto-failover (standby host)
• vSphere HA
• SAP HANA system replication

Disaster recovery includes:

• Backup and restore
• Storage replication
• vSphere Replication
• SAP HANA system replication

We include SAP HANA system replication (HSR) in both recovery scenarios. When site or data recovery is required, you can use HSR as a failover or disaster recovery solution. HSR reduces startup time for large SAP HANA databases by preloading data into the replication instance's memory.

You can assign different recovery point objectives (RPOs) and recovery time objectives (RTOs) to different fault recovery and disaster recovery solutions. SAP describes the phases of high availability in their HANA HA document. The following figure shows these phases.

Figure 33. SAP HANA system availability phases

• RPO (1 – possible data loss) specifies the amount of possible data that can be lost due to a failure. It is the time between the last valid backup and/or last available SAP HANA save point, and/or the last saved transaction log file that is available for recovery and the point in time of the error situation. All changes made during this time may be lost and are not recoverable.
• Detect (2 – time needed to detect a failure) shows the time needed to detect a failure and start the recovery steps. This is usually done in seconds. vSphere HA tries to automate the detection of a wide range of error situations, thus minimizing the detection time.
• RTO, recover (3 – time needed to recover failure) is the time needed to recover from a fault. Depending on the failure, this may require restoring a backup or restarting the SAP HANA processes.
• Performance ramp (4 – time needed to resume full operation at a defined SLA) shows the performance ramp, which describes the time needed for a system to run at the same service level as before the fault (data consistency and performance).

Based on this information, you can plan and implement the proper HA/recovery solution to meet the customer-specific RPOs and RTOs.

The goal of the IT team operating SAP HANA should be to minimize RTOs and RPOs with the available IT budget and resources. VMware virtualized SAP HANA systems allow this by highly standardizing and automating the failure detection and recovery process.

About vSphere high availability (HA)

VMware provides vSphere built-in and optional availability and disaster recovery solutions to protect a virtualized SAP HANA system at the hardware and operating system levels. Many of the key features of virtualization, such as encapsulation and hardware independence, already offer inherent protections. Furthermore, vSphere can provide fault tolerance by incorporating redundant components like dual network and storage pathing, supporting hardware solutions like UPS, and incorporating built-in CPU features that can withstand memory model failures and ensure CPU transaction consistency.

These features are available on the ESXi host without needing to be configured on the VM or application. Organizations can meet their RPOs and RTOs with additional protections like vSphere HA.

VMware Site Recovery Manager can manage and automate various HA solutions, from component-level failures to complete site failures, as depicted in the figure below. These features protect any application running inside a VM against hardware failures, allow planned maintenance with zero downtime, and protect against unplanned downtime and disasters.

Figure 34. VMware HA and DRS solutions provide protection at every level

vSphere HA is a fault recovery solution that provides uniform, cost-effective failover protection against hardware and operating system outages within a virtualized IT environment. It does this by monitoring ESXi hosts and VMs to detect hardware and guest failures. When it detects a server outage, it automatically restarts VMs on other ESXi hosts in the cluster, thereby reducing application downtime. This, combined with the SAP HANA service auto-restart feature, allows HA levels of 99.9% out of the box ( EMC IT, 02/14 EMC Perspective,  H12853).

The next figure shows how vSphere HA can protect against VM or host failure. Application protection solutions—such as SAP HANA service auto-restart, third-party in-guest cluster solutions, and SAP HANA system replication—can also provide disaster recovery capabilities. You can combine all these solutions with vSphere HA.

Figure 35. Virtualized SAP HANA HA solution

vSphere HA safeguards SAP HANA scale-up and scale-out deployments without requiring any external components, like DNS servers, or solutions like the SAP HANA Storage Connector API.

The following figure shows how critical SAP HANA VMs and non-SAP HANA VMs that run on different, lower-cost server systems can leverage vSphere HA and how a typical n+1 vSphere cluster can be configured to survive a complete host failure.

SAP applications and SAP HANA instances on vSphere use this vSphere HA configuration. If you need higher redundancy levels, you can use an n+2 configuration. You can also use the HA resource pool for non-critical VMs, but you must power them off before restarting an SAP HANA or SAP app server VM.

Note: The following figures don't show the vCLS control plane or the related vCLS.

Figure 36. vSphere HA-protected SAP HANA VMs in an n+1 cluster configuration

It's also possible to configure the HA cluster as active-active, where all hosts have SAP HANA VMs deployed. This guarantees the utilization of all hosts in a vSphere cluster, ensuring sufficient failover capacity for all running VMs in the event of a host failure. The arrow in the figure indicates that the VMs can failover to different hosts in the cluster. This active-active cluster configuration assumes that one host's capacity (n+1) is always available to support a complete host failure.

As previously mentioned, you can also use vSphere HA to protect an SAP HANA scale-out deployment. Unlike a physical scale-out deployment, SAP HANA on vSphere can withstand a host failure without the need for dedicated standby hosts and storage-specific implementations.

There are no dependencies on external components, such as DNS servers, the SAP HANA Storage Connector API, or STONIT scripts. vSphere HA will simply restart the failed SAP HANA VM on the vSphere HA/standby server. The HANA VM mounts the HANA shared directory via NFS, just like physical systems do. Access to the HANA shared directory is therefore guaranteed. If the NFS server providing this share is also virtualized, you can use vSphere Fault Tolerance (FT) to protect it.

The following figure shows a configuration of 3 SAP HANA 4-socket VMs (one leader with two follower nodes) running exclusively on the host of a vSphere cluster based on 4-socket hosts. One host provides the needed failover capacity for a host failure.

Figure 37. SAP HANA scale-out on a vSphere HA n+1 cluster configuration

It's possible to use the HA node for other workloads while in normal operation. If you use the HA/standby node for other workloads, you must stop or migrate all VMs running on this host before you can restart a failed HANA scale-out VM on the host. In this case, the overall failover time could be a bit longer because, if configured correctly, vSphere HA will wait until all needed resources are available on the failover host.

You can use up to 16 scale-out nodes with up to 3TB on a 4-socket VM or 6TB on an 8-socket VM. SAP HANA on vSphere supports only 4- and 8-socket host systems with Intel Broadwell or newer CPUs. SAP doesn't support 2-socket host systems for on-premises scale-out deployments. Review the previously mentioned SAP HANA on vSphere support notes for more details.

In physical SAP HANA deployments, SAP HANA system replication is the only method to provide fault recovery. Implementing a third-party solution like SUSE HA is necessary to automate the recovery. Protecting a physical SAP HANA deployment against host failures is, therefore, relatively complex. In contrast, protecting an SAP HANA on vSphere system can be done with just a mouse click.

If you require fast failure recovery and data replication, we recommend you use HANA System Replication (HSR) in combination with a SAP-supported Linux cluster solution like Pacemaker. Because HSR replicates SAP HANA data, you can use it for disaster recovery or for recovering from logical errors (depending on the log retention policy).

vSphere HA with passthrough network adapters

If the VMXNET3-caused latency is too high for a specific use case/workload, we recommend you use a passthrough NIC configuration.

You must configure vSphere HA with passthrough NICs as a dynamic vSphere DirectPath I/O device, assigning it a unique cluster-wide hardware label. To do this, follow the instructions in Add a PCI Device to a Virtual Machine. You must use the same hardware label for the passthrough NIC installed in the HA host. If you haven't configured an HA host with a passthrough NIC as a dynamic vSphere DirectPath I/O device and the same hardware label exists, then the HA failover process won’t work. For more information about this topic, read Assignable Hardware.

SAP HANA system replication with vSphere (local site)

You can combine vSphere HA with HANA System Replication (HSR) to protect SAP HANA data against logical or disastrous failures that impact a data center.

In this case, vSphere HA would protect against local failures, such as OS or local component failures, and HSR would protect the SAP HANA data against logical or data center failures. HSR requires a running SAP HANA replication VM, which must receive HSR data. Alternatively, you can use storage subsystem-based replication, which would be independent of SAP HANA.

The next figure shows a vSphere cluster with an SAP HANA production VM replicated to an SAP HANA replication VM. The HSR replica VM can be running on the same cluster or on another vSphere cluster/host to protect against data center failures. If it runs in the same location, you can use HSR to recover from logical failures (if log collection is delayed) or to reduce the ramp-up time of an SAP HANA system because data can already be loaded into the replication server's memory. HSR can change direction depending on which HANA instance is the production one.

Figure 38. vSphere HA with local data and HANA system replication (HSR)

As noted, HSR doesn't provide automated failover. You must manually reconfigure the replication target VM to match the production system’s identity. Alternatively, you can use 3^rd-party cluster solutions such as SAP HANA HA Linux solutions from SUSE or Red Hat or SAP Landscape Management to automate the failover to the SAP HANA replication target.

To provide disaster tolerance, you must place the HSR replica VM/host on another data center or even a geographically dispersed site.

Disaster recovery

You can combine the discussed HA solutions with storage/vSphere Replication and HSR to another data center/site if data protection is also a priority or if a complete site failover of all IT systems, including SAP HANA, is necessary.

In addition to this, you also need backup and restore solutions in place for regulatory reasons and to protect the data against logical errors.

SAP HSR and vSphere Replication for a remote site

The figure below shows an HSR-protected SAP HANA instance. The concept remains unchanged from the previous figure, except for the placement of the HSR replication target in a different data center. This provides additional protection against data center failures or, if the remote data center is on another site, it protects against site failures. The ESXi host in DC-2 can be either a standalone one or a member of a vSphere cluster. Stretched vSphere clusters are also possible and supported.

Some replication requirements (synchronous or asynchronous) might call for a roundtrip time (RTT) below 1 millisecond to maintain the SAP HANA storage KPIs. If a 1 millisecond RTT is not possible, then you should use asynchronous replication to ensure that the data replication doesn't negatively impact the production SAP HANA system.

Figure 39. vSphere HA with remote data center and SAP HANA system replication (HSR)

The figure above shows HSR replication from one virtualized SAP HANA system to another. You can leverage vSphere Replication to replicate the SAP app server VMs. This will allow the operation to continue after switching to this data center. These app servers can also run on dedicated non-SAP HANA host systems.

vSphere replication is a hypervisor-based, asynchronous replication solution for vSphere VMs (VMDK files). It allows recovery point objective (RPO) times ranging from 5 minutes to 24 hours. The VM replication process is nonintrusive and takes place independently of the VM's OS or applications. It is transparent to protected VMs and requires no changes to their configuration or ongoing management.

If they're not using HSR, admins often use vSphere Replication with clusters stretched over two separate data centers to protect their systems against local data center failures. If you're using this method, and your systems should also be protected against data center site disasters, then you need to replicate all relevant systems to a second site. As previously mentioned, you can do so using vSphere Replication, native storage replication, and SAP HANA system replication.

vSphere Replication operates at the individual VMDK level, allowing individual VMs to replicate between heterogeneous storage types that vSphere supports. Because vSphere Replication is independent of the underlying storage, it works with a variety of storage types, including vSAN, vSphere Virtual Volumes (vVols), traditional SAN, network-attached storage (NAS), and direct-attached storage (DAS).

If you use an SAP HANA HCI based on vSAN solution, data center distances of up to 5 kilometers/3 miles are supported.

Backing up and restoring SAP HANA systems virtualized on vSphere

Backing up and restoring an SAP HANA database and the Linux VM supporting SAP HANA is the same as backing up bare-metal deployed SAP HANA systems.

The simplest approach involves performing a file system backup along with a HANA database dump, which you can conveniently run within SAP HANA Studio. If you are using a backup solution, you can leverage the backint interface. Refer to the SAP HANA product documentation for backup and restore information and requirements.

In addition, any SAP- and VMware-supported backup solution that leverages SAP HANA studio/backint with vSphere snapshots can protect a vSphere-deployed SAP HANA system (refer to the next figure). This provides a neutral storage vendor backup solution based on vSphere snapshots, resulting in shorter backup and restore times. For an example, refer to the Veeam backup and recovery solution designed for vSphere and SAP HANA.

Caution: Using vSphere snapshot backup solutions not integrated with SAP HANA can lead to complications. These issues arise because the VM must be frozen to create a snapshot while SAP HANA is running. For this reason, we advise you to use VMware snapshots only when SAP HANA has been stopped or when all IO activities have been paused to ensure data consistency.

Figure 40. Virtualized SAP HANA backup and recovery methods

SAP HANA with persistent memory on vSphere 7

Prerequisites and general SAP support limitations for Intel Optane Persistent Memory (PMem)

• Review information published in SAP note 2913410: SAP HANA on VMware vSphere with Persistent Memory.
• Only server platforms that are SAP HANA-supported and certified and listed on the VMware Compatibility Guide with the Persistent Memory feature are supported.
• Review the VMware/Intel PMem required SAP HANA best practices.
• The CPU and memory prerequisites of SAP HANA on vSphere must be met. For details, refer to SAP notes 2937606 and 3102813.
• General questions and boundary conditions about SAP HANA and Persistent Memory are addressed in SAP notes 2700084 and 2786237, and Broadcom KB 78094.
• For SAP HANA non-production support for memory mode, see SAP note 2954515: SAP HANA Persistent Memory - memory mode.

What is supported?

SAP has granted support for SAP HANA 2 SPS 4 (or later) on vSphere 7.0 (beginning with version 7.0 P01) for 2- and 4-socket servers based on second-generation Intel Xeon scalable processors (formerly code-named Cascade Lake). PMem doesn't support 8-socket hosts. The maximum DRAM plus PMem host memory configurations with SAP HANA support 4-socket VMs on 4-socket hosts and can be up to 15TB (the current memory limit when DRAM is combined with PMem) and must follow the hardware vendor’s PMem configuration guidelines.

The maximum VM size with vSphere 7.0 is limited to 256 vCPUs and 6TB of memory. This results in SAP HANA VM sizes of 6 TB maximum for OLTP and 3 TB VM sizes for OLAP workloads (class L). vSphere 7.0 supports OLAP workloads up to 6 TB (class M). The supported DRAM to PMem ratios are 2:1, 1:1, 1:2, and 1:4. Please refer to SAP note 2700084 for further details, use cases, and assistance in determining whether PMem can be used for your specific SAP HANA workload.

Supported PMem module sizes are 128GB, 256GB, and 512GB. The following table lists the maximum supported host memory DRAM and PMem configurations. Up to two SAP HANA VMs are supported per CPU socket, and up to 4-socket ESXi hosts can be used. See the table below for the currently supported configurations.

[1] vSphere 7.0 U2 or later versions are required for VM sizes >6 TB.

Sizing of Optane PMem-enabled SAP HANA VMs

PMem-enabled SAP HANA VM sizings are like those for bare-metal SAP HANA systems, but are limited to a maximum size of 6TB (mix of DRAM and PMem) per VM. OLAP class-L workload sizings are limited to 3TB. Class-M sizings support up to 6TB total memory.

Please refer to SAP notes 2700084 and 2786237: Sizing SAP HANA with Persistent Memory for details on compute and memory sizing for SAP HANA systems using PMem.

We recommend that an SAP HANA VM use the same DRAM to PMem ratio as the physical host/server DRAM to PMem ratio. If you have a growth plan, you might consider a larger physical memory configuration and upgrade the VMs and SAP HANA over your environment's lifetime.

For example, let's say you have a 1:4 PMem ratio host configured with 15TB of total RAM (3TB DRAM and 12TB PMem). To optimize resources, you'd create 4 SAP HANA VMs on this server, each with 3.75TB RAM (0.75TB DRAM and 3TB PMem). If you created 6TB VMs on the same 15TB host, you could only create 2 SAP HANA VMs. This results in unoptimized resources because your system can only utilize 12TB of the installed 15TB memory. In this case, a 1:1 DRAM to PMem configuration with a total of 12TB (6TB DRAM and 6TB PMem) would optimize the configuration.

Figure 41. Non-optimized host memory configuration

4-socket host configuration:

Four 2nd Gen Intel Xeon Platinum processors

24 x 128 GB DRAM + 24 x 512GB PMem = 15TB total host memory with a 1:4 DRAM to PMem ratio

VM configuration example:

2 x 6TB SAP HANA VM with 1.5TB DRAM and 4.5TB PMem RAM, with a 1:3 DRAM to PMem ratio

Challenges:

• DRAM: The PMem ratio may not be suited for the SAP HANA workload
• The hardware configuration does not fit and will lead to unusable PMem (a ratio mismatch)

Figure 42. Optimized host memory configuration

4-socket host configuration:

Four 2nd Gen Intel Xeon Platinum processors

24 x 256 GB DRAM + 24 x 256GB PMem = 12TB total host memory with a 1:1 DRAM to PMem ratio

VM configuration example:

2 x 6 TB SAP HANA VM with 3TB DRAM and 3TB Optane PMem RAM, with a 1:1 DRAM to PMem ratio

Challenges:

• Higher memory costs due to DRAM module prices

Figure 43. Optimized host memory configuration

4-socket host configuration:

Four 2nd Gen Intel Xeon Platinum processors

24 x 128GB DRAM + 24 x 256GB PMem = 9TB total host memory

VM configuration example:

4 x VM with 0.75TB DRAM and 1.5TB PMem RAM = total RAM per SAP HANA VM 2.25TB with a 1:2 DRAM to PMem ratio

Challenges:

• Need to verify if the PMem ratio is applicable to the SAP HANA sizing and if the CPU resources are enough.

Because PMem in App Direct mode provides data persistence in memory and is local to the host, not all vSphere features can be equally used on a DRAM-only VM. See the following table for details.

Using SAP HANA on vSphere allows HANA users to leverage the flexibility of vSphere capabilities, such as vMotion, which allows workloads to be migrated between ESXi hosts on Intel Xeon platforms without first having to be shut down. In addition, vSphere DRS works with a cluster of ESXi hosts to provide resource management capabilities, such as load balancing and VM placement, to ensure a balanced environment for VM workloads.

vSphere HA is supported for SAP HANA VMs with PMem use cases. For more information, read the VMware blog post, VMware vSphere 7.0 U2 and vSphere HA for SAP HANA with DRAM and Intel Optane PMem in App-Direct Mode.

Table 33. vSphere features supported with PMem-enabled SAP HANA VMs

vSphere HA support for PMem-enabled SAP HANA VMs

Before the vSphere 7.0 U2 release, vSphere HA did not support PMem-enabled VMs. vSphere HA now supports failing over and restarting PMem-enabled VMs. Applications using PMem must maintain data persistence on PMem and shared disks.

SAP HANA is one of the applications that provides data persistence on disk. Because of this, vSphere HA can use the data on shared disks to initiate a failover of PMem-enabled SAP HANA VMs to another PMem host. vSphere HA will automatically recreate the VM’s NVDIMM configuration, but it doesn't have control over post-VM failover OS/application-specific configuration steps, such as the required re-creation of the SAP HANA DAX device configuration. This must be done manually or via a script, that is not provided by VMware or SAP. For details on how to configure PMem for SAP HANA, see the Intel Optane Persistent Memory and SAP HANA Platform Configuration guide.

The figure below illustrates the failover of a PMem-enabled SAP HANA VM via vSphere 7.0 U2 and vSphere HA, and highlights that the PMem NVDIMM configuration is automatically re-created as part of the VM failover process. Once the DAX device is configured inside the OS, you can start SAP HANA. It will automatically load the data from the disk to the new PMem regions assigned to this VM.

Figure 44. vSphere HA support for PMem-enabled VMs

After a successful failover of this PMem-enabled VM, a garbage collector process will identify failed-over VMs and free up the PMem resources previously used by this VM on the initial host. On the host this VM now runs on, the PMem will be blocked and reserved for the lifetime of this VM (as long as it isn't migrated or deleted from the host).

The Intel SAP Solution Engineering team, as well as the Intel and VMware Center of Excellence, have developed an example script for automatically re-creating the DAX device configuration at the OS level. Run this script after the VM has failed over and restarted but before the SAP HANA database has been restarted. It's best to automatically run this script as part of the OS start procedure, such as a custom service. You can use the script as a template to create your own script that fits your unique environment.

SAP HANA with PMem VM configuration details

Using PMem in a vSphere virtualized environment requires that the physical host, ESXi, and VM configurations are correctly configured.

Follow the Intel Optane Persistent Memory and SAP HANA Platform Configuration on VMware ESXi configuration guide to prepare the needed DAX devices and see how to configure SAP HANA to enable PMem.

The following list outlines the configuration steps. Refer to the hardware vendor–specific documentation to correctly configure PMem for SAP HANA.

Host:

1. Configure the server host for PMem in its BIOS (vendor specific)
2. Create AppDirect interleaved regions and verify that they are configured for ESXi use.

VM:

1. Create a VM with hardware version 19 (vSphere 7.0 U2 or later) with NVDIMMs and allow failover to another host while doing this.
2. Edit the VMX VM configuration file and make the NVDIMMs NUMA aware.

Operating system:

1. Create a file system on the namespace (DAX) devices in the OS.
2. Configure SAP HANA to use the persistent memory file system.
3. Restart SAP HANA to activate and start using PMem.

Details on configuration steps 2 and 3

1. Before you can add NVDIMMs to an SAP HANA VM, check if the PMem regions and namespaces were created correctly in the BIOS.
    
2. Also, make sure you selected all PMem as persistent memory and that the persistent memory type is set to App Direct Interleaved, as shown in the following figure.
   
   Figure 45. Example of PMem system BIOS settings
   
    
3. After creating the PMem memory regions, you must reboot the system.
    
4. Install the latest ESXi version (7.0 U2 or later) and, using the web client, check if the PMem memory modules, interleave sets, and namespaces have been set up correctly. See the examples in figures 46–49.
   
   Figure 46. ESXi Persistent Memory Storage view of Modules
   
   
   Figure 47. ESXi Persistent Memory Storage view of interleave sets
   
   
   Figure 48. ESXi Persistent Memory Storage view of Namespaces
   
   
   
   Note: The interleave set numbers shown depend on the hardware configuration and may differ in your configuration.
   
   If the configurations is done correctly in the host's BIOS, it should look like what is shown in these figures. 
    
5. After this, you can add NVDIMMs and NVDIMM controllers to your SAP HANA VM. Select the maximum size possible per NVDIMM; otherwise, you'll waste memory capacity.
   
   Figure 49. The vSphere Client Edit settings screen, where you add NVDIMMs and NVDIMM controllers to the SAP HANA VM
   
   

To configure a PMem-enabled SAP HANA VM for optimal performance, you must align the VM configuration to the underlying hardware, especially the NUMA configuration. Broadcom KB 78094 provides information on how to configure the NVDIMMs (VMware’s representation of Optane PMem) correctly and align the NVDIMMs to the physical NUMA architecture of the physical server.

By default, the PMem allocation in vmkernel for VM NVDIMMs does not consider NUMA. This can result in the VM running on a certain NUMA node and the PMem being allocated on a different NUMA node. This will cause the NVDIMMs to access the VM remotely instead of locally, resulting in poor performance. To solve this, you must add the following settings to a VM configuration using vCenter.

Example for a 4-socket wide VM:

nvdimm.mode = "independent-persistent"
nvdimm0:0.nodeAffinity=0
nvdimm0:1.nodeAffinity=1
nvdimm0:2.nodeAffinity=2
nvdimm0:3.nodeAffinity=3
sched.pmem.prealloc=TRUE (optional)

Note: sched.pmem.prealloc=TRUE is an optional parameter equivalent to eager zero thick provisioning of VMDKs and improves initial writes to PMem. Be aware that the first vMotion process with this parameter set will take a long time due to the pre-allocation of PMem in the target server.

Besides these parameters, you can configure the CPU NUMA node affinity or CPU affinities (pinning) as described in the SAP HANA best practices parameter guidelines listed in the Best practices of virtualized SAP HANA systems section.

Note: The parameters in the example above must be manually added after the PMem SAP HANA VM is created.

Verify the VMX file of the newly created VM and check if the NVDIMM configuration looks like the following example. The easiest way to do this is to use the ESXi PowerShell.

Example output of the .vmx file of a PMem-enabled VM:

[root@ESXiHOSTxxx:/vmfs/volumes/xxxxxx/PMem_SAP_HANA_VM_name] grep -i nvdimm *.vmx" nvdimm0:0.present = "TRUE"
nvdimm0:0.fileName = "/vmfs/volumes/pmem:XXXXX/ PMem_SAP_HANA_VM_name_1.vmdk" nvdimm0:0.size = "757760"
nvdimm0:1.present = "TRUE"
nvdimm0:1.fileName = /vmfs/volumes/pmem:XXXXX/ PMem_SAP_HANA_VM_name_3.vmdk" nvdimm0:1.size = "757760"
nvdimm0:2.present = "TRUE"
nvdimm0:2.fileName = /vmfs/volumes/pmem:XXXXX/ PMem_SAP_HANA_VM_name_5.vmdk" nvdimm0:2.size = "757760"
nvdimm0:3.present = "TRUE"
nvdimm0:3.fileName = /vmfs/volumes/pmem:XXXXX/ PMem_SAP_HANA_VM_name_5.vmdk" nvdimm0:3.size = "757760"
nvdimm0:0.node = "0"
nvdimm0:1.node = "1"
nvdimm0:2.node = "2"
nvdimm0:3.node = "3"

Manually added parameters:

• nvdimm.mode = "independent-persistent"
• nvdimm0:0.nodeAffinity=0
• nvdimm0:1.nodeAffinity=1
• nvdimm0:2.nodeAffinity=2
• nvdimm0:3.nodeAffinity=3
• sched.pmem.prealloc=TRUE (optional and will cause time delays during the first vMotion process)

Monitoring and verifying an SAP HANA installation

SAP notes for monitoring data growth/CPU utilization and verifying configuration

• SAP note 1698281 provides information about how you can monitor the data growth and the utilization of actual memory. With this, it is also possible to detect and diagnose the memory leaks during operation.
• SAP note 1969700 covers all the major HANA configuration checks and presents a tabular output with changed configurations. Collect SQL statements to check and identify parameters that are configured  and conflict with the SAP-recommended configuration parameters.

NUMA tool: VMware NUMA Observer

While you may configure large critical VMs with affinities to unique logical cores or NUMA nodes, maintenance, and HA events can change this unique mapping. An HA event would migrate VMs to other hosts with spare capacity and those hosts may already be running VMs affined to the same cores or sockets. This results in multiple VMs constrained/scheduled to the same set of logical cores. These overlapping affinities may result in a CPU contention and/or non-local allocation of memory.

To check if the initial configuration is correct or to detect misalignments, you can download and use the NUMA Observer fling (downloads a .zip file).

NUMA Observer scans your VM inventory and identifies VMs with overlapping core/NUMA affinities and generates alerts. Additionally, the fling also collects statistics on remote memory usage and CPU starvation of critical VMs and raises alerts, see figures 50 and 51 for examples.

Figure 50. VMware NUMA Observer – VM Core Overlap Graph

Figure 51. VMware NUMA Observer – VM Alerts

Performance optimizations for SAP HANA VMs

This section discusses the parameters needed to optimally configure an SAP HANA VM. The most critical aspect of these optimizations is properly aligning NUMA nodes to get the best performance and the lowest memory latency.

Optimizing the configuration parameter list

VMware vSphere can run a single large SAP HANA VM or multiple smaller VMs on a single physical host. This section describes how to optimally configure a VMware virtualized SAP HANA environment. These parameters are valid for SAP HANA VMs running vSphere and vSAN-based SAP HANA HCI configurations.

The listed parameter settings are the recommended BIOS settings for the physical server, the ESXi host, the VM, and the Linux OS to achieve optimal operational readiness and stable performance for SAP HANA on vSphere.

The parameter settings described in this section are the default settings that should always be configured for SAP HANA on vSphere. The settings described in the Performance optimization for low-latency SAP HANA VMs section should be applied only in rare situations where SAP HANA must perform with the lowest latency possible.

The shown parameters are best practice configurations, and, in case of an escalation, the support engineers will verify and, if not applied, they will recommend configuring these settings.

The following table shows the sub-NUMA node cluster (SNC) settings that are required when a half-socket VM runs on a 2-socket Sapphire Rapids ESXi host.

Table 34. Physical host BIOS parameter setting

Table 35. ESXi host parameter settings

Table 36. SAP HANA VM parameter settings

Table 37. Sub NUMA Node Cluster (SNC) settings

numa.vcpu.preferHT="TRUE"
sched.node0.affinity ="0"

numa.vcpu.preferHT="TRUE"
sched.node0.affinity ="1"

numa.vcpu.preferHT="TRUE"
sched.node0.affinity ="2"

numa.vcpu.preferHT="TRUE"
sched.node0.affinity ="3"

numa.vcpu.preferHT="TRUE"
sched.node0.affinity ="0"
sched.node1.affinity ="1"

numa.vcpu.preferHT="TRUE"
sched.node0.affinity="2"
sched.node1.affinity="3

numa.vcpu.preferHT="TRUE"
sched.node0.affinity="1"
sched.node1.affinity="2"

numa.vcpu.preferHT="TRUE"
sched.node0.affinity="0"
sched.node1.affinity="1"
sched.node2.affinity="2"
sched.node3.affinity="3"

Table 38. Linux operating system parameter settings

options vmw_pvscsi cmd_per_lun=254 ring_pages=32

vmw_pvscsi.cmd_per_lun=254
vmw_pvscsi.ring_pages=32

ethtool -K ethY lro off

Table 39. General SAP HANA Linux configuration recommendations (These settings will be automatically configured when you use SAPTune/SAPConf.)

numa_balancing=disabled

etc/sysctl .d/sap_hana.conf

kernel.numa_balancing = 0

# sysctl -p /etc/sysctl.d/sap_hana.conf 

# cat/sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

# echo never > /sys/kernel/mm/ transparent_hugepage/enabled 

net.core.rmem_default = 262144
net.core.wmem_max = 8388608
net.core.wmem_default = 262144
net.core.rmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.ipv4.tcp_slow_start_after_idle = 0

intel_idle.max_cstate=0 processor.max_cstate=0 numa_balancing=disabled transparent_hugepage=never elevator=noop vmw_pvscsi.cmd_per_lun=254 vmw_pvscsi.ring_pages=32

Performance optimization for low-latency SAP HANA VMs

When SAP HANA must perform as clost to bare metal as possible with the lowest database access latency, you'll need to optimize the performance of SAP HANA VMs. We recommend sizing SAP HANA VMs with the fewest NUMA nodes for low latency. Configure an additional NUMA node and its resources for SAP HANA VMs that need more CPU or RAM.

To achieve optimal performance for an SAP HANA VM, use the settings as described in table 40 in addition to the previously described settings. CPU scheduling and priority settings improve performance by reducing the amount of vCPU and vNUMA migration, while increasing the priority of the SAP HANA production VM.

CPU affinity settings

By specifying a CPU affinity setting for each VM, you can restrict the assignment of VMs to a subset of the available processors (CPU cores) in multiprocessor systems. By using this feature, you can assign each VM to processors in the specified affinity set.

Setting CPU affinities can decrease the CPU and memory latency by preventing the ESXi scheduler from migrating VM threads to other logical processors. Setting CPU affinities is required when configuring SAP HANA half-socket VMs.

Before you use a CPU affinity, consider the following:

• For multiprocessor systems, ESXi systems perform automatic load balancing. Avoid manually setting VM affinity to allow the scheduler to optimally balance load across processors.
• An affinity setting can interfere with the ESXi host’s ability to meet the reservation and shares specified for a VM.
• Because CPU admission control does not consider affinities, a VM with manual affinity settings might not always receive its full reservation. VMs that do not have manual affinity settings are not adversely affected by ones with manual affinity settings.
• When you move a VM from one host to another, a manually set affinity might no longer apply because the new host might have a different number of processors.
• The NUMA scheduler might not be able to manage a VM that is already assigned to the processor using a manually set affinity.
• An affinity setting can affect the host’s ability to schedule VMs on multicore or hyperthreaded processors, preventing the host from fully taking advantage of the resources shared on those processors.

For more information about performance practices, see the vSphere Resource Management Guide and the VMware documentation on specifying NUMA controls.

Additional performance tuning settings for SAP HANA

Table 40. Tunings for very low-latency SAP HANA VMs

monitor.idleLoopSpinBeforeHalt = "true" 
monitor.idleLoopMinSpinUS = "xx us"

monitor.idleLoopSpinBeforeHalt = "true"
monitor.idleLoopMinSpinUS = "xx" 

monitor_control.halt_in_monitor = "TRUE"

monitor_control.halt_in_monitor = "TRUE"

monitor_control.disable_pause_loop_exiting = "TRUE"

Table 41. Settings to improve NUMA alignment

sched.vcpuXx.affinity = "Yy-Zz"

numa.slit.enable = "TRUE" 

Examples of VMX configurations for SAP HANA VMs

The following examples provide an overview of how to set additional VMX parameters for SAP HANA half- and full-CPU socket VMs. These parameters can be added through the vSphere Web Client or by directly adding these parameters to the .vmx file with a text editor.

Table 42. Additional VMX parameters to set for half-socket SAP HANA VMs on 28-core/n-socket CPU server; socket 0

numa.vcpu.preferHT="TRUE"
sched.vcpu0.affinity = "0-27"
sched.vcpu1.affinity = "0-27"
sched.vcpu2.affinity = "0-27"
…
sched.vcpu26.affinity = "0-27"
sched.vcpu27.affinity = "0-27"

nvdimm0:0.nodeAffinity=0
numa.vcpu.preferHT="TRUE"
sched.vcpu0.affinity = "0-27"
sched.vcpu1.affinity = "0-27"
sched.vcpu2.affinity = "0-27"
…
sched.vcpu26.affinity = "0-27"
sched.vcpu27.affinity = "0-27"

numa.vcpu.preferHT="TRUE"
sched.vcpu0.affinity = "28-55"
sched.vcpu1.affinity = "28-55"
sched.vcpu2.affinity = "28-55"
…
sched.vcpu26.affinity = "28-55"
sched.vcpu27.affinity = "28-55"

nvdimm0:0.nodeAffinity=1
numa.vcpu.preferHT="TRUE"
sched.vcpu0.affinity = "84-111"
sched.vcpu1.affinity = "84-111"
sched.vcpu2.affinity = "84-111"
…
sched.vcpu26.affinity = "84-111"
sched.vcpu27.affinity = "84-111"

Table 43. Additional VMX parameters to set for single-socket SAP HANA VMs on a 28-core/8-socket CPU server; 
socket 3

numa.vcpu.preferHT="TRUE"
sched.node0.affinity="3"

nvdimm0:0.nodeAffinity=3
numa.vcpu.preferHT="TRUE"
sched.node0.affinity="3"

Table 44. Additional VMX parameters to set for dual-socket SAP HANA VMs on a 28-core/n-socket CPU server; 
sockets 0-1

numa.vcpu.preferHT="TRUE"
sched.node0.affinity="0"
sched.node1.affinity="1"

nvdimm0:0.nodeAffinity=0
nvdimm0:1.nodeAffinity=1
numa.vcpu.preferHT="TRUE"
sched.node0.affinity="0"
sched.node1.affinity="1"

Table 45. Additional VMX parameters to set for 3-socket SAP HANA VMs on a 28-core/4- or 8-socket CPU server; 
sockets 0-2

numa.vcpu.preferHT="TRUE"
sched.node0.affinity="0"
sched.node1.affinity="1"
sched.node2.affinity="2"

nvdimm0:0.nodeAffinity=0
nvdimm0:1.nodeAffinity=1
nvdimm0:2.nodeAffinity=2 
numa.vcpu.preferHT="TRUE"
sched.node0.affinity="0"
sched.node1.affinity="1"
sched.node2.affinity="2"

Table 46. Additional VMX parameters to set for 4-socket SAP HANA VMs on a 28-core/4-socket CPU

nvdimm0:0.nodeAffinity=0
nvdimm0:1.nodeAffinity=1
nvdimm0:2.nodeAffinity=2
nvdimm0:3.nodeAffinity=3

Table 47. Additional VMX parameters to set for 4-socket SAP HANA VM on a 28-core/8-socket CPU; sockets 0-3

numa.slit.enable = "TRUE"
sched.vcpu0.affinity = "0-55"
sched.vcpu1.affinity = "0-55"
sched.vcpu2.affinity = "0-55"
…
sched.vcpu222.affinity = "168-223"
sched.vcpu223.affinity = "168-223"

Table 48. Additional VMX parameters to set for 6-socket SAP HANA VM on a 28-core/8-socket CPU server; sockets 0-5

numa.slit.enable = "TRUE"
sched.vcpu0.affinity = "0-55"
sched.vcpu1.affinity = "0-55"
sched.vcpu2.affinity = "0-55"
…
sched.vcpu334.affinity = "280-335"
sched.vcpu335.affinity = "280-335"

Table 49. Additional VMX parameters to set for 8-socket SAP HANA VM on a 28-core/8-socket CPU server

numa.slit.enable = "TRUE"
sched.vcpu0.affinity = "0-55"
sched.vcpu1.affinity = "0-55"
sched.vcpu2.affinity = "0-55"
…
sched.vcpu446.affinity = "392-447"
sched.vcpu447.affinity = "1392-447"

Table 50. Additional VMX parameters to set for an SAP HANA VM running a latency-sensitive application

monitor.idleLoopSpinBeforeHalt = "TRUE"
monitor.idleLoopMinSpinUS = "50"

monitor_control.disable_pause_loop_exiting = "TRUE"

CPU thread matrix examples

Figure 52 shows the CPU thread matrix of a 28-core CPU as a reference when configuring the sched.vCPUXx.affinty = "Xx-Yy" parameter. The list shows the start and end ranges required for the "Xx-Yy" parameter (for example, for CPU 5, this would be 280–335).

Figure 52. CPU thread examples for a 28-core/56-thread CPU server

Figure 53 shows the CPU thread matrix of a 24-core CPU as a reference when configuring the sched.vCPUXx.affinty ="Xx-Yy" parameter. The list shows the staring and end range required for the "Xx-Yy" parameter. For example, for CPU 5 it would be "240-287".

Figure 53. CPU thread examples for a 24-core/48-thread CPU server

 Figure 54 shows the CPU thread matrix of a 22-core CPU as a reference when configuring the sched.vCPUXx.affinty ="Xx-Yy" parameter. The list shows the staring and end range required for the "Xx-Yy" parameter. For example, for CPU 5 it would be "220-263".

Figure 54. CPU thread examples for a 22-core/44-thread CPU server

SAP HANA support and process

To get support for your virtualized SAP HANA systems, open a ticket directly with SAP. The ticket will be routed directly to VMware and SAP HANA support engineers, who will then troubleshoot the escalated issue.

Open an SAP support request ticket

VMware is part of the SAP support organization, allowing VMware support engineers to work directly with SAP, SAP customers, and other SAP software partners, such as SUSE, as well as with hardware partners, on solving issues that need escalation.

Before opening a VMware support ticket, we recommend opening a support request within the SAP support system for SAP HANA systems virtualized with vSphere. This ensures that SAP HANA and VMware specialists will work on the case and, if needed, escalate the issue to VMware product support (when it is a VMware product issue) or to SAP support (when it is an SAP HANA issue).

The following components are available for escalating SAP on vSphere issues:

• BC-OP-NT-ESX (Windows on VMware ESXi)
• BC-OP-LNX-ESX (Linux on VMware ESXi and SAP HANA)

You should escalate issues related to SAP HANA on vSphere to BC-OP-LNX-ESX using the SAP Solution Manager. For non-vSphere-related SAP HANA issues, the escalation will be moved to the correct support component. Figure 55 shows the support process workflow for vSphere-related SAP HANA issues.

Figure 55. SAP support workflow for vSphere-related escalations

If the problem cannot be solved by the first and second levels of SAP support, they forward it to the next support level.

For example, if the issue is a Linux kernel panic or an SAP HANA product issue, we recommend that you use the correct support component instead of using VMware support because this may delay the support process. If you are uncertain that the issue is related to vSphere, open the ticket first at the general SAP HANA support component.

If the issue is related to vSphere, such as an ESXi driver, then you may either open the ticket via SAP Solution Manager and escalate it to BC-OP-LNX-ESX or ask the VMware customer administrator to open a support ticket directly at VMware.

Open a VMware support request ticket

If there appears to be a VMware product issue or if vSphere is not configured optimally and is causing a bottleneck, file a support request using VMware Customer Connect at http://www.vmware.com/support/contacts/file-sr.html.

In addition:

• Follow the troubleshooting steps outlined in the VMware knowledge base article, Troubleshooting ESX/ESXi virtual machine performance issues (2001003).
• Run the vm-support utility, and then run the following command at the service console: 
  vm support-s
  
  This command collects the necessary information that VMware uses to help diagnose issues. It is best to run this command when symptoms first occur.

If you want to escalate an issue with your SAP HANA HCI solution, please work directly with your HCI vendor and follow the defined and agreed support process, which normally starts by opening a support ticket within the SAP support tools and selecting the HCI partners SAP support component.

Conclusion

SAP HANA on VMware vSphere/VMware Cloud Foundation provides a cloud operation model for your business-critical enterprise application and data.

For over 12 years, virtualizing SAP HANA with vSphere has been supported. Compared to a natively installed SAP HANA system, virtualizing SAP HANA with vSphere does not require any specific considerations for deployment and operation.

In addition, your SAP HANA environment gains all the virtualization benefits of easier operation, such as SAP HANA database live migration with vMotion or strict resource isolation on a virtual serve, increased security, standardization, better service levels and resource utilization, an easy HA solution using vSphere HA, lower total cost of ownership, an easier way to maintain compliance, faster time to value, reduced complexity and dependencies, custom SAP HANA system sizes optimally aligned to your workload and needs, and the previously mentioned advantage of using a cloud operation model.

I think anything "software-defined" means it’s digital. It means we can automate it, and we can control it, and we can move it much faster.

—Andrew Henderson, Former CTO, ING Bank

About the Author

Erik Rieger is a Principal SAP Global Technical Alliance Manager & Architect, working with VMware by Broadcom's Global SAP Alliance. He defines and manages the VMware SAP solution and validation roadmap. Erik works closely with SAP, partners, and customers to define the architectures of VMware-based SAP solutions so that they have the best mix of functionality to help transform businesses into real-time enterprises. Erik has more than 20 years of experience in the IT sector, has a technical degree in electronics, and a Master of Science degree in Information Systems and Management.

Acknowledgments

The following individuals from VMware by Broadcom contributed content or helped review this guide:

• Fred Abounader, Staff Performance Engineer, Performance Engineering team
• Louis Barton, Staff Performance Engineer, Performance Engineering team
• Julie Brodeur, Senior Technical Writer, Performance Engineering team
• Pascal Hanke, Solution Consultant, Professional Services team
• Sathya Krishnaswamy, Staff Performance Engineer, Performance Engineering team
• Sebastian Lenz, Staff Performance Engineer, Performance Engineering team
• Todd Muirhead, Staff Performance Engineer, Performance Engineering team
• Oliver Ziltener, Senior Technical Product Manager, VCF division
• Catherine Xu, Manager of Workload, Technical Marketing team