SAP HANA on Hyperconverged Infrastructure (HCI) Solutions Powered by VMware vSAN

Abstract

This document provides best practices and architecture guidelines to design and build an SAP HANA environment on Hyperconverged Infrastructure (HCI) solutions based on Skylake, Cascade Lake, and Cooper Lake) (second generation of INTEL® XEON® SCALABLE processor) based systems running with VMware vSphere® and VMware vSAN™. 

The document provides information any hardware vendor can use as a starting point to build their own unique SAP HANA HCI solutions.

The vendor is responsible to define and select the best vSAN ready hardware components, VMware vSphere, and vSAN configuration for its unique solution. The HCI solution vendor is also responsible to certify its solution accordingly the SAP HANA Hardware Certification Hyperconverged Infrastructure certification scenario “HANA-HyCI 1.0 v1.2” in the SAP Integration and Certification Center (ICC).

This paper also includes information about the support processes and requirements of SAP HANA on vSAN.

The new document version 2.1 of this document includes now support for 8-socked HCI host servers and stretched cluster configurations.

Executive Summary

Overview

For more than 10 years, VMware has successfully virtualized SAP applications on x86 server hardware. It is a logical next step to virtualize and use software to define the storage for these applications and their data. SAP has provided production support for their applications on VMware vSphere for the last 10 years, and for HANA since 2014.

SAP HANA based applications require dynamic, reliable, and high-performing server systems coupled with a predictable and reliable storage subsystem. The servers and storage must deliver the needed storage throughput and IOPS capacity at the right protection level and costs for all SAP HANA VMs in the environment.

Storage solutions like large Fibre Channel SAN or standalone SAP HANA appliance configurations, are not able to quickly react to today’s changing and dynamic environment. Their static configurations have difficulty keeping pace with modern and resource-demanding applications.

VMware vSAN, a software-defined storage solution, allows enterprises to build and operate highly flexible and application-driven storage configurations, which meets SAP HANA application requirements. Detailed SAP support information for VMware vSAN can be found in SAP Note 2718982 - SAP HANA on VMware vSphere and vSAN. A valid SAP s-user is required to view the note.

Audience

This paper is intended for SAP and VMware hardware partners who want to build and offer SAP HANA certified HCI solutions based on VMware technologies.

Solution Overview

SAP HANA on vSAN

SAP HANA is an in-memory data platform that converges database and application platforms, which eliminates data redundancy and improves performance with lower latency. SAP HANA architecture enables converged, online transaction processing (OLTP) and online analytical processing (OLAP) within a single in-memory, column-based data store that adheres to the Atomicity, Consistency, Isolation, Durability (ACID) compliance model.

SAP environments with SAP HANA can use vSAN running on certified x86 servers to eliminate traditional IT silos of compute, storage, and networking with this hyperconverged solution.

All intelligence and management are moved into a single software stack, allowing for a VM- and application-centric policy-based control and automation.

This brings better security, predictable performance, operational simplicity, and cost-effectiveness into SAP HANA environments. vSAN also simplifies the complexity of traditional IT silos for compute, storage, and networking, as it allows managing the SAP HANA storage just like compute resources.

HCI lowers costs by combining the economics and simplicity of local storage with the features and benefits of shared storage and can get used for SAP HANA Scale-Up and Scale-Out deployments.

Following figure summarizes the benefits of VMware based SPA HCI solutions.

Only SAP and VMware certified and tested components, like server and storage systems, are supported. These components must comply with the SAP HANA Hardware Certification Hyperconverged Infrastructure (HANA-HyCI) test scenario.

Note: Only SAP and VMware HCI solution vendor offered solutions, that get SAP HANA HCI solution certified, are supported for SAP HANA workloads. SAP applications other than HANA are fully supported on a VMware vSAN cluster. No additional support notes or specific support processes are required from SAP when running other applications on vSAN, like an SAP application server.

Solution Components

An SAP HANA HCI solution based on VMware technologies is a fully virtualized and cloud ready infrastructure turnkey solution running on VMware vSphere and vSAN. All components like compute, storage, and networking are delivered together as an HCI vendor specific SAP HANA solution. It is based on a tightly integrated VMware software stack, where vSAN is part of the VMware ESXi™ kernel to efficiently provide great performing storage.

There are two ways to build a vSAN SAP HANA HCI cluster for an OEM:

  • Turnkey deployment using the existing vSAN/VCF appliances and modify these systems to be SAP HANA HCI ready.
  • Use the certified vSAN Ready Nodes and components and build an SAP HANA ready HCI solution. 

The solution consists of the following components:

  • VMware certified vSAN HW partner solutions, as listed on the VMware vSAN HCL (vSAN Appliances or Ready Nodes with vSAN Ready Components).
  • SAP HANA supported server systems, as listed on the SAP HANA HCL.
  • VMware vSphere Enterprise Plus starting with version 6.5 update 2. For Cascade Lake based servers vSphere 6.7u3 or later and for Cooper Lake based servers vSphere 7.0 U2 is required. Review SAP note 2393917 for an updated list of supported vSphere versions.
  • VMware vSAN Enterprise starting with version 6.6. For Cascade Lake based servers vSAN 6.7u3 or later is required. Review SAP note 2718982 for an updated list of supported vSAN versions.
  • VMware vCenter 6.5, 6.7 U3, 7.0 U1 and U2. Note that vCenter corresponding versions are required.
  • HCI vendor implemented the vSAN HCI detection script provided by VMware (adds vSAN HCI identifier to the VM configuration file, see the SAP HANA HCI VMware vSAN Detection Script section for details).
  • HCI vendor SAP-integrated support process.

After the solutions are defined and built, SAP ICC validates and lists these solutions in Certified HCI Solutions on the SAP HANA Supported and Certified Hardware Directory section.

Figure 1 provides an overview of the solution components.

Figure 1.                      SAP HANA HCI Solution Overview

SAP HANA HCI on vSAN—Support Status as of Note 2718982

SAP HANA HCI solutions are currently supported with the following VMware vSphere and vSAN versions and hardware:

  • VMware vSphere Hypervisor 6.5, 6.7, 6.7 U2/U3,  7.0 U1 and 7.0 U2
    • SAP HANA SPS 12 Rev. 122.19 (or later releases) for production single-VM and multi-VM use cases. See SAP Note 2393917 for detailed information and constraints. SAP HANA on vSAN has additional constraints, see next section.
    • vSphere 7.0 U1 and U2 are supported as of today for SAP HANA HCI solutions based on the vSphere 7.0 resource maximums, which are 6 TB RAM and up to 256 vCPUs per VM.
  • VMware vSAN 6.6, 6.7 (incl. U1, U2, and U3), 7.0 U1 and 7.0 U2[1]
    • vSAN 6.6, 6.7, 6.7 U1, 6.7 U2, 6.7 U3, 7.0 U1, 7.0 U2: Maximum 4 SAP HANA VMs on 2, 4, and 8 socket servers
    • Starting from above vSAN versions, the HCI vendor can select the appropriate vSAN version for their HCI solution.
    • The vSAN versions 6.6, 6.7, 7.0 U1, and 7.0 U2 are included in vSphere version 6.5 U2 and vSphere version 6.7, 6.7 U2/U3, 7.0 U1, and 7.0 U2 respectively.
    • Minimal 3, maximal 64 nodes per SAP HANA HCI vSAN cluster.
    • 4 or more nodes are recommended for optimal failure resilience, even during maintenance and for cross data center wide, HA solutions are required.
    • No CPU sockets need to get reserved for vSAN. Note that vSAN will need, just like ESXi, system resources. Refer to VMware KB 2113954 for information about vSAN resource needs.
  • CPU architecture and VM maximums:
    • Any supported Intel Skylake, Cascade Lake or Cooper Lake processors with 2, 4, and 8 socket servers with existing VMware vSphere virtualization boundaries, for example:
      • max. 6 TB RAM and 256 vCPUs per SAP HANA Scale-Up VM, see SAP Note 2393917 for details. Also note that the actual usable vCPUs depend on the used CPU type: the current vSphere maximum vCPU number is 256.
    • The certified hardware for HCI solutions can be found on the Certified Hyper-Converged Infrastructure Solutions
  • SAP HANA Deployment Options:
    • SAP HANA Scale-Up:
      • OLTP or mixed type workload up to 6 TB RAM and up to 256 vCPUs per VM
      • OLAP type workload:
        • SAP sizing Class L: up to 3 TB RAM and up to 256 vCPUs per VM
        • SAP sizing Class M: up to 6 TB RAM and up to 256 vCPUs per VM
    • SAP HANA Scale-Out:
      • Up to 1 master and 7 worker nodes are possible, plus HA node(s). HA nodes should be added according to the cluster size. FAQ: SAP HANA High Availability
      • SAP HANA Scale-Out system max. 16 TB memory with SAP HANA CPU L-Class* sizing:
        • Up to 2 TB of RAM and up to 256 vCPUs per Scale-Out node (VM), sizing up to SAP HANA CPU Sizing Class L type workloads
      • SAP HANA Scale-Out system max. 24 TB memory with SAP HANA M-Class* CPU sizing:
        • Up to 3 TB of RAM and up to 256 vCPUs per Scale-Out node (VM), sizing up to SAP HANA CPU Sizing Class M** type workloads
      • Additional requirements:
        • Scale-Out is only support by SAP with 4 or 8 socket servers and minimal 4-socket wide VMs that use up to 256 logical CPU threads, in the case of Cascade Lake 28 core CPUs this represents a maximal VM size with 224 vCPUs.

Note: SAP Note 2718982 - SAP HANA on VMware vSphere and vSAN will be continuously updated with the latest support information without further notice. Above support status is from September 2020. Before planning an SAP HANA HCI vSAN based solution, check note 2718982 for any updates. 

The SAP HANA BW needed CPU Sizing Class can be found in the BW sizing report under the heading "RSDDSTAT ANALYSIS DETAILS". See SAP note 2610534 for details.

Reference Architecture

The purpose of the introduced vSAN for SAP HANA reference architectures is to provide a starting point for HCI solution vendors who want to build a vSAN based solution and to provide best practices and guidelines.

The reference architecture is tested and validated with the SAP specified HCI tests and is known to be good with the shown SAP HANA configuration.

Scope

This reference architecture:

  • Provides a technical overview of an SAP HANA on a VMware HCI solution
  • Shows the components used to test and validate vSAN with the SAP HCI tests and to meet the storage performance requirements of SAP HANA Tailored Datacenter Integration (TDI)
  • Explains configuration guidelines and best practices
  • Provides information about vSAN/HCI solution sizing for SAP HANA VMs
  • Explains the deployment options from single DC to multiple suite configurations

SAP HANA HCI on vSAN—Architecture Overview

vSAN Clusters consist of two or more physical hosts that contain either a combination of magnetic disks and flash devices (hybrid configuration) or all-flash devices (all-flash configuration). All vSAN nodes contribute cache and capacity to the vSAN distributed/clustered datastore and are interconnected through a dedicated 10 GbE (minimum) or better path redundant network.

Note: In SAP HANA environments, four nodes are recommended for easier maintenance and Stretched Cluster readiness, but the minimum requirement of a three-host vSAN cluster for SAP HANA HCI can be used.

Figure 2 shows the high-level architecture of an SAP landscape running on a VMware-based SAP HANA HCI solution built upon several vSphere hosts and vSAN for a single data center configuration. Unlike with other SAP HANA HCI solutions, a VMware vSAN powered SAP HANA HCI solution can leverage all available ESXi host CPU and RAM resources. As shown in the figure, one host has enough compute resources available to support a complete host failover.

Figure 2.                      SAP HANA Applications on vSAN High-Level Architecture

Storage policies for less demanding SAP HANA application servers and for IO demanding HANA VMs, as shown in Figure 2 (bronze, silver, and gold), will ensure that all VMs get the performance, resource commitments, failure tolerance, and quality of service as a required policy-driven dynamic environment. With such an environment, changing the assigned storage policy in response to the changing needs of a virtualized application can directly influence the underlying infrastructure. The different storage policies allow changing the level of protection or possible IO limits without disturbing operation of a VM, allowing it to remain available and maintain the defined SLAs and performance KPIs.

SAP HANA HCI on vSAN—Scalability and Sizes

SAP HANA on vSphere is supported across a range of SAP HANA system size. The smallest size is a half socket CPU configuration with a minimum of 8 physical CPU cores and 128 GB of RAM. The largest size, as of today, is a 4 socket VM with 6 TB of RAM and 256* vCPUs.  Scale-Out SAP HANA deployments are also supported with up to eight 3 TB nodes. In total, up to 24 TB system memory can be deployed on SAP HANA vSAN HCI solutions for OLAP type workloads.

vSphere 6.7 U3, 7.0 U1 and U2* as supported for vSAN provides the following maximums per physical host (table 1). For SAP HANA, only servers with up to 8 physical CPU sockets are currently supported.

*Actual usable number of vCPU depends on the selected CPU type. vSphere 7.0 U1 supports up to 6 TB and 256 vCPUs.

Table 1.  Current vSphere HCI Supported Host Maximums

 

ESX 6.7 U3 / 7.0

ESX 7.0 U2

Logical CPUs per host

768

896

Virtual machines per host

1024

Virtual CPUs per host

4096

Virtual CPUs per core

32

RAM per host

16 TB

24 TB

NUMA Nodes / CPU sockets per host

16 (SAP HANA only 8 CPU sockets)

Table 2 shows the maximum size of an SAP HANA VM and some relevant parameters such as virtual disk size and number of virtual NICs per VM. The current VM size is limited for SAP HANA to 224 vCPUs and 6 TB of RAM (4-socket wide VM).

Table 2.  vSphere Guest (VM) Maximums

 

ESX 6.7 U3 / 7.0

ESX 7.0 U2

Virtual CPUs per virtual machine

256

768

RAM per virtual machine

6128 GB

24 TB

Virtual CPUs per HANA virtual machine

224

224

RAM per HANA virtual machine

6128 GB

6128 GB

Virtual SCSI adapters per virtual machine

4

Virtual NVMe adapters per virtual machine

4

Virtual disk size

62TB

Virtual NICs per virtual machine

10

Table 3 shows the vSAN maximum values: the component number or the size of a vSAN SAP HANA HCI cluster depends on the SAP HANA HCI vendor. The minimum SAP HANA supported cluster configuration is 3 vSAN cluster nodes, and up to 64 cluster nodes are supported by vSAN.

Table 3.  vSAN Maximums

VMware vSAN 6.7 (incl. U3) and 7.0 U1/U2

vSAN ESXi host

vSAN Cluster

vSAN Virtual Machines

vSAN disk groups per host

5

 

 

Cache disks per disk group

1

 

 

Capacity tier maximum devices per disk group

7

 

 

Cache tier maximum devices per host

5

 

 

Capacity tier maximum devices

35

 

 

Components per vSAN host

9,000

 

 

 

 

 

 

Number of vSAN hosts in an All-Flash cluster

 

64

 

Number of vSAN hosts in a hybrid cluster

 

64

 

Number of vSAN datastores per cluster

 

1

 

 

 

 

 

Virtual machines per host

 

 

200

Virtual machine virtual disk size

 

 

62 TB

Disk stripes per object

 

 

12

Percentage of flash read cache reservation

 

 

100

Percentage of object space reservation

 

 

100

vSAN networks/physical network fabrics

 

 

2

Table 4 shows the size of SAP HANA VMs possible on a supported SAP HANA HCI solution for production use case with vSphere and vSAN 6.7 U3, 7.0 U1 and U2.

As of SAP note 2393917, the maximum size of a single SAP HANA VM is 6 TB of RAM and 224 vCPUs for Cascade Lake based server systems vCPUs. 256 vCPUs are the current vSphere maximum. SAP HANA VM can be configured as half-socket, full-socket, 2, 3, or 4-socket wide VMs. No CPU sockets need to be reserved for vSAN. If an SAP HANA system fits into these maximums, it can be virtualized with vSphere in a single VM. If it is larger, in terms of CPU or RAM needs, Scale-Out or SAP HANA data-tiering solutions can be used alternatively to meet the requirements.

We have validated that 4 SAP HANA VMs can work consolidated on a vSAN configuration running on 2- or more-socket systems. For detailed information, see the Reference Test System Configuration section.

Table 4 shows the example CPU’s VM sizes that can be deployed on a vSAN SAP HANA HCI node.

Note: RAM must be reserved for ESXi. The RAM figures listed in Error! Reference source not found. show the theoretical RAM maximums of a VM. The RAM for ESXi needs to get subtracted from this RAM size. No CPU sockets or other system resources need to get reserved for vSAN. VMware KB 2113954 provides details about the vSAN and memory consumption in ESXi.

Table 4.  Example SAP HANA VM Configurations and Capacity of VMware Based HCI Solutions with the selected 2nd Generation INTEL® XEON® Scalable Processors

 

Intel Xeon Gold 6258R CPU

Intel Xeon Platinum 8280L CPU

Intel Xeon Platinum 8380HL CPU

SAP Benchmark

2020015 (05. May 2020)

2019023 (02. April 2019)

2020050 (11. Dec. 2020)

Max supported RAM per CPU as of Intel datasheet.

1024 GB

4.5 TB

4.5 TB

CPU Cores per socket as of Intel datasheet.

28

28

28

Max. NUMA Nodes per ESXi Server

2

82

82

vSAPS per CPU thread w/ and w/o HT

2459 (core w/o HT)

434 (HT gain)

Based on SD benchmark cert. 2020015

2596 (core w/o HT)

458 (HT gain)

Based on SD benchmark cert. 2019023

2432 (core w/o HT)

429 (HT gain)

Based on SD benchmark cert.2018020

0.5 Socket SAP HANA VM (Half-Socket)

1 to 4 x 14 physical core VM with min. 128 GB RAM and max. 512GB3

1 to 16 x 14 physical core VM with min. 128 GB RAM and max. 768 GB3

1 to 16 x 14 physical core VM with min. 128 GB RAM and max. 768 GB3

vSAPS4 34,000, 28 vCPUs

vSAPS4 36,000, 28 vCPUs

vSAPS4 34,000, 28 vCPUs

1 Socket SAP HANA VM

1 to 2 x 28 physical core VM with min. 128 GB RAM and max. 1024 GB

1 to 8 x 28 physical core VM with min. 128 GB RAM and max. 1536 GB

1 to 8 x 28 physical core VM with min. 128 GB RAM and max. 1536 GB

vSAPS4 81,000, 56 vCPUs

vSAPS4 85,000, 56 vCPUs

vSAPS4 80,000 56 vCPUs

2 Socket SAP HANA VM

1 x 56 physical core VM with min. 128 GB RAM and max. 2048 GB

1 to 4 x 56 physical core VM with min. 128 GB RAM and max. 3072 GB

1 to 4 x 56 physical core VM with min. 128 GB RAM and max. 3072 GB

vSAPS4 162,000,112 vCPUs

vSAPS4 171,000, 112 vCPUs

vSAPS4 160,000, 112 vCPUs

3 Socket SAP HANA VM

-

1 to 2 x 84 physical core VM with min. 128 GB RAM and max. 4608 GB

1 to 2 x 84 physical core VM with min. 128 GB RAM and max. 4608 GB

-

vSAPS4 256,000, 168 vCPUs

vSAPS4 240,000, 168 vCPUs

4 Socket SAP HANA VM

-

1 to 2 x 112 physical core VM with min. 128 GB RAM and max. 6128 GB

1 to 2 x 112 physical core VM with min. 128 GB RAM and max. 6128 GB

-

vSAPS4 342,000, 224 vCPUs

vSAPS4 320,000, 224 vCPUs

1SAP supports in appliance configuration up to 15 TB for Suite and 1 TB for BW SAP HANA applications with SKL and later CPU’s. More RAM may be possible with a TDI phase 5 / workload-based sizing.

2Maximal 224 vCPUs can get configured per VM with vSphere 6.7 U2 and later with a Skylake, Cascade Lake or Cooper Lake CPU.

3The half-socket RAM figures listed in the table show even configured half-socket VM CPU configurations and RAM sizes.

4The listed vSAPS figures are based on published SD benchmark results with Hyperthreading (2-vCPU configuration) and minus 10% virtualization costs. In the case of a half-socket configuration in addition to the 10% virt. costs, 15% from the SD capacity must get subtracted. The shown figures are rounded figures and based on rounded SAPS performance figures published SAP SD benchmarks, and can get only used for Suite or BW on HANA or BW/4HANA workloads. For mixed HANA workloads sizing parameters contact SAP or you HW vendor.

In addition to Scale-Up SAP HANA system RAM sizes of up to 6 TB, Scale-Out deployments are supported with up to 3 TB VMs, distributed over up to 8 SAP HANA HCI plus HA nodes as required. This provides for an OLAP type SAP HANA instance up to 24 TB in total system RAM by utilizing the Scale-Out deployment model.

Figure 3 shows a 4+1 SAP HANA HCI BW cluster. As of today, up to eight[2] 3 TB Scale-Out nodes (VMs) are supported with vSAN. The required SAP HANA “Shared Directory” gets mounted directly to SAP HANA VMs via NFS from an SAP HANA HCI external NFS server.

A screenshot of a cell phone</p>
<p>Description automatically generated

Figure 3.                      SAP HANA Scale-Out Example Cluster on vSAN (HCI servers)

VMware HA

VMware vSphere HA delivers a vSphere built-in high availability solution, which can be used to protect SAP HANA. VMware HA provides unified and cost-effective failover protection against hardware and operating system outages and works with a vSAN data store just like it does with any other supported shared storage solution with VMFS. These options are for single site.

In addition, if there is a disaster recovery site, SAP HANA instances can run across site on a stretched cluster.

VMware HA to Protect Scale-Up and Scale-Out SAP HANA Systems

VMware HA protects SAP HANA Scale-Up and Scale-Out deployments without any dependencies on external components, such as DNS servers, or solutions, such as the SAP HANA Storage Connector API or STONITH scripts.

Note: See the Architecture Guidelines and Best Practices for Deployments of SAP HANA on VMware vSphere for detailed information about how VMware HA is used to protect any SAP HANA deployment on vSphere and how to install an SAP HANA Scale-Out system on VMware vSphere.

In the event of a hardware or OS failure, the entire SAP HANA VM is restarted either on the same or other hosts in the vSphere cluster. Because all virtual disks, such as OS, data, and log VMDKs are stored on the shared vSAN datastore, no specific storage tasks, STONITH scripts, or extra cluster solutions are required.

Figure 4 shows the concept of a typical n+1 configuration, which is the standard deployment of an SAP HANA HCI solution. It is recommended deploying an n+2 configuration for easy maintenance and to protect your systems against a failure while you perform maintenance.

Figure 4.                      VMware HA Protected SAP HANA VMs in an n+1 vSAN Example Cluster Configuration

Figure 5 shows the same configuration with an active-active cluster. This ensures that all hosts of a vSAN cluster are used by still providing enough failover capacity for all running VMs in the case of a host failure.

Figure 5.                      VMware HA Active-Active n+1 vSAN Example Cluster Configuration

The same concept is also used for SAP HANA Scale-Out configurations. Note that for SAP HANA Scale-Out, only 4-socket or 8-socket servers are supported. The minimum SAP HANA VM must be configured with 4 virtual CPU sockets respectively. Figure 6 shows a configuration of 3 SAP HANA VMs (1 x master with 2 worker nodes) running on the virtual SAP HANA HCI platform. The NFS mounted SAP HANA shared directory is not part of the vSAN cluster and needs to be provided as an vSAN external NFS service[*].

Figure 6.                      SAP HANA Scale-Out VMware HA n+1 vSAN Example Cluster Configuration

Unlike physical deployments, there are no dependencies on external components, such as DNS servers, SAP HANA Storage Connector API, or STONIT scripts. VMware HA will simply restart the failed SAP HANA VM on the VMware HA/standby server. HANA shared is mounted via NFS, just as recommended with physical systems and will move with the VM that has failed. The access to HANA shared is therefore guaranteed.

Note: VMware HA can protect only against OS or VM crashes or HW failures. It cannot protect against logical failures or OS file system corruptions that are not handled by the OS file system. vSAN 7.0 U1 File Service (NFS) are yet not supported.

VMware vSAN Stretched Cluster Support SAP HANA Systems

Stretched clusters extend the vSAN cluster from a single data site to two sites for a faster level of availability and cross-site load balancing. Stretched clusters are typically deployed in environments where the distance between data centers is limited, such as metropolitan or campus environments. SAP HANA is now tested and is supported on vSAN stretched cluster 7.0 U1/U2 environments. SAP HANA requires that log file writes have a less than 1ms latency, and this log file write latency should not exceed 1ms across primary and secondary sites. The underlying network infrastructure plays a key role, as the distances supported between the two sites solely depends on ability to achieve latency of 1ms across these sites for SAP HANA.

Tests conducted with one of our HCI hardware partners showed that distances up to 30 km between the two sites, all SAP HANA performance throughput and latency KPIs were passed across the tested SAP HANA VMs. Actual distances could be lower since the network latency is depending on the used network components and architecture. SAP HANA HCI support of stretched cluster configurations is upon the SAP HANA HCI hardware vendor and not VMware.

You can use stretched clusters to manage planned maintenance and avoid disaster scenarios, because maintenance or loss of one site does not affect the overall operation of the cluster. In a stretched cluster configuration, both data sites are active sites. If either site fails, vSAN uses the storage on the other site. vSphere HA restarts any VM that must be restarted on the remaining active site.

You must designate one site as the preferred site. The other site becomes a secondary or nonpreferred site. If the network connection between the two active sites is lost, vSAN continues operation with the preferred site. The site designated as preferred typically is the one that remains in operation unless it is resyncing or has another issue. The site that leads to maximum data availability is the one that remains in operation.

Below example shows a simplified stretched cluster configuration. Site X is the preferred (prod. site), which can be a complete datacenter or a single rack. Site Y is the secondary site and in this case the failover site in the case of site failure. Local failures can get compensated by providing HA capacity inside the site and a site failover can get initiated by providing enough failover capacity at site Y, or by freeing up failover capacity. The witness appliance must run on a host placed in an independent site Z. The witness decides ultimately on a site failover and can run for instance on an ESXi host which supports the SAP application server VMs.

 

Screen Shot 2021-09-16 at 3.50.02 PMFigure 7.                      SAP HANA Stretched Cluster Example Configuration

Summary:

  • Tested distances maintain <= 1 ms KPI for SAP HANA log writing up to 30 km
  • Stretched clusters are normally used campus wide, therefore distances up to 5 km supportable.
  • Witness VM can run on any vSphere host in 3rd size “z”.
  • Failover capacity is n+1, where 1 stands for one host failure. N+2 would support two host failures.
  • Failover can be bi-directional.

SAP HANA Stretched Clusters VM Storage Policy

For an SAP HANA VM in vSAN stretched cluster, the VM storage policy should be enhanced for Dual Site Mirroring and with Failures to tolerate equal to 1 failure – RAID1 (Mirroring) as shown in figure 8.

Figure 8.                      SAP HANA VM Storage Policy in vSAN Stretched Cluster

When Site Disaster Tolerance is set to “Dual Site Mirroring”, a copy of the data should go to both sites, When the “Primary Failures to Tolerate” rule is equal to 1, writes will continue to be written in a mirrored fashion across sites, therefore check and ensure that the required disk capacity is available on each site.

Reference Test System Configuration

This section introduces the resources and configurations including:

  • Reference environment
  • Quality of service policy
  • Hardware resources
  • Software resources

Note: The shown reference configuration should be treated as an example but not as a requirement. The HCI solution vendor may or may not follow the shown example configurations and select different components. The only requirements are that the SAP HANA HCI KPIs are met and that only supported/certified components are used.

Reference Environment

To ensure that a vSAN based HCI solution is SAP HANA ready, SAP provided test tools and test cases (HANA-HyCI 1.0 v1.2 test scenario). These test tools validate performance, scalability, and consistency. Under the “Certified Hyper-Converged Infrastructure Solutions“, a complete set of vSAN supported configurations from different vendors are listed. For example, this link provides a list for Cascade Lake and vSAN supported configurations.

The reference configurations listed on the next pages are validated by VMware and provide an example for Scale-Up and Scale-Out configurations. The Scale-Up example configurations are based on 2-,4 and 8-socket systems, configured as a 3-host and 4-host vSAN clusters with 2 disk groups and 4 disk groups per server. The configurations had Intel Optane NVMe devices as caching layer, and SSD based capacity disks. With both configurations, it is possible to meet the SAP HANA KPIs defined for each configuration. The Scale-Out test configuration ran on a 4-host vSAN cluster. Each host had 3 TB memory, 4 CPU sockets, and 4 disk groups installed and configured. In the 8-socked host case two individual Scale-Out systems could get consolidated on an 8-socket host configuration.

IOPS Limit per Partition Policy—Quality of Service Option to Storage Throughput

vSAN storage policies provide multiple options, many features are built into storage policy that has capability to limit the IOPS on the partition that this policy is enforced upon. This IOPS value can be set when a policy is created. Typically, a value of “0” ensures that there is no IOPS limit on partition that this policy is enforced upon. After the policy is created, it can be applied to any partition or partitions across VMs or single VM depending on requirements.

A typical HANA instance has log and data partition. We have done internal testing scenarios where the IOPS limit is set on data partition only, across a set of 8 HANA VMs RAID 1, running on a 3-host cluster with one HA host. To do this, configure a policy with limits and allocate this policy on VMDK which will be mounted as data partition. A minimum setting of around 13,700 IOPS on the test setup's data partition, has shown capability to handle the KPI requirements of data partition. This is the minimum value and anything above this value is also valid. However, for your configuration, it is advisable to do a sanity check with around above ballpark value. For log partition, it is not recommended to set any limits at this time.

Use cases :

1. Depending on application requirements, such a policy will ensure that there is fair IO distribution among different VMs for that specific partition. This is an option that customers can use, if there are multiple HANA VMs running, if there are multiple HANA VMs running, which requires control to ensure a minimum IO distribution among HANA VMs.

2. Other use cases are that you would want to control the IOPS done by other VMs but want to have one main database VM (unlimited IOPs) as the one getting most of IO resources.

Hardware Resources

Two vSAN clusters were configured in the 4-host and 3-host environment respectively:

  • One vSAN SAP HANA HCI environment with four 2-socket based Cascade Lake processors with 2 disk groups per each vSAN host (example 1).
  • One vSAN SAP HANA HCI environment with 4 hosts, having a combination of three 4 socket Cascade Lake and one 2 socket Cascade Lake processors configured with 4 disk groups per each vSAN host (example 2).
  • One vSAN SAP HANA HCI environment with 3 hosts, having a combination of two 8 socket Cascade Lake and one 4 socket Cascade Lake processors configured with 3 disk groups per each vSAN host (example 3).

The idea using different text configuration is to test a variant of system configurations and to provide to the SAP HANA HCI hardware vendors a wide range of proven test configurations they can choose from when building their own solutions.

2-Socket/2-Disk Group SAP HANA HCI vSAN Based—Example 1

The 2-socket/2-disk group test environment had four vSphere ESXi servers configured and used direct-attached SAS and NVMe devices inside the hosts to provide a vSAN datastore that meets the SAP HANA HCI test requirements and KPIs for two SAP HANA VMs per server.

The fourth node was part of the vSAN datastore and provided HA capabilities for the total 6 running SAP HANA VMs in the vSAN cluster.

Each vSphere ESXi server had two disk groups each consisting of one 1.46 TB NVMe device for caching and four capacity tier 3.49 TB SAS SSDs. The raw capacity of the vSAN datastore of this 4-node vSAN cluster was for data around 112 TB. The vSAN cluster limit is up to 64 cluster nodes. The vSAN cluster node number that an HCI vendor supports is up to the vendor.

Two 25 Gbit network adapters were used for vSAN and another network traffic in active-standby failover manner.

Each ESXi Server in the vSAN Cluster had the following configurations as shown in the following table, based on this configuration, up to two running SAP HANA VMs per HCI server.

Property

SPECIFICATION

ESXi server model

HW Vendor SAP HANA Supported Cascade Lake Server Model

ESXi host CPU

2 x Intel® X®(R) Gold 6248 CPU, 2 Sockets, 20 Physical Cores/Socket.

ESXi host RAM

1.5 TB

Network adapters

2 x 25 Gbit as a shared network for vSAN, SAP application and admin network traffic.

See the network section of this guide for the recommended network configurations. SAP HANA HCI Network Configuration

Storage adapter

4 x 8 channel PCI-EXPRESS 3.0 SAS 12GBS/SATA 6GBS SCSI controller

Storage configuration

Disk Groups: 2

Cache Tier: 2 x 750 GB Intel Optane DC 4800X Series

Capacity Tier: 12 X 3.8 TB 3840 Micron 5200 TCG-E

Linux LVM configuration

Used Linux LVM for log and data.

Used 4 VMDK disks in LVM each (total 8) to create the needed log and a data volume for SAP HANA.

 

4-Socket/4-Disk Group SAP HANA HCI vSAN Cluster—Example 2

The 4-socket test environment had three vSphere ESXi servers configured and used direct-attached SSDs and NVMe inside the hosts to provide a vSAN datastore that meets the SAP HANA HCI test requirements and KPIs.

The third node was part of the vSAN datastore and provided HA capabilities for the total 8 running SAP HANA VMs. Adding the fourth node would allow you to run more SAP HANA VMs on the vSAN datastore. The vSAN cluster limit is up to 64 cluster nodes. The vSAN cluster node number that an HCI vendor supports is determined by the vendor.

Each vSphere ESXi server had four disk groups with each consisting of 800 GB NVMe devices for caching and four capacity tier 890GB SAS SSDs. The raw capacity of the vSAN datastore of this 3-node vSAN cluster was for data around 43 TB. The vSAN cluster limit is up to 64 cluster nodes.

Two 25 Gbit network adapters were used for vSAN and other network traffic in active-standby failover manner.

Review the network section for details about how the network cards impact the number of SAP HANA VMs per host and the sizing section for how many SAP HANA VMs can be deployed on a Skylake based server system.

Each ESXi Server in the vSAN Cluster had the following configuration as shown in Table 5. This configuration supports up to four SAP HANA VMs per HCI server.

Table 5.  ESXi Server Configuration of the 4-socket Server SAP HANA on 3-node HCI vSAN Cluster

PROPERTY

SPECIFICATION

ESXi server model

HW Vendor SAP HANA Supported Cascade Lake supported Model

ESXi host CPU

4 x Intel® Xeon(R) Platinum 8280M CPU 2.70GHz , 4 Sockets, 28 Physical Cores/Socket

ESXi host RAM

1.5 TB

Network adapters

2 x 25 Gbit as a shared network for vSAN, SAP application and admin network traffic.

See the network section of this guide for recommended network configurations.

Storage adapter

SAS 12GBS/SATA 6GBS SCSI controller

Storage configuration

Disk Groups: 4

Cache Tier: 4 x Express Flash PM1725a 800GB SFF

Capacity Tier: 16 x  SAS  894 GB SSD

8-Socket/4-socket 3 Disk Group SAP HANA–HCI vSAN Cluster—Example 3

The 8-socket and 4-socket test environment had a total of three vSphere ESXi servers configured and used direct-attached SAS SSDs and NVMe cache drives inside the hosts to provide a vSAN datastore that meets the SAP HANA HCI test requirements and KPIs.

The third node was part of the vSAN datastore and provided HA capabilities for the total 8 running SAP HANA VMs. Adding the fourth node would allow you to run more SAP HANA VMs on the vSAN datastore. The vSAN cluster limit is up to 64 cluster nodes. The vSAN cluster node number that an HCI vendor supports is determined by the vendor.

Each vSphere ESXi server had three disk groups with each consisting of 3TB NVMe devices for caching and four capacity tier 1.6TB SAS SSDs. The raw capacity of the vSAN datastore of this 3-node vSAN cluster was for data around 58 TB. The vSAN cluster limit is up to 64 cluster nodes.

Two 25 Gbit network adapters were used for vSAN and other network traffic in active-standby failover manner.

Review the network section for details about how the network cards impact the number of SAP HANA VMs per host and the sizing section for how many SAP HANA VMs can be deployed on a Skylake based server system.

Each ESXi Server in the vSAN Cluster had the following configuration as shown in Table 6. This configuration supports up to four SAP HANA VMs per HCI server.

Table 6.  ESXi Server Configuration of the 4-socket Server SAP HANA on 3-node HCI vSAN Cluster

PROPERTY

SPECIFICATION

ESXi server model

HW Vendor SAP HANA Supported Cascade Lake supported Model

ESXi host CPU

Intel® Xeon®Platinum 8276L CPU @ 2.2GHz, 28 cores,

2 hosts of 8 socket

1 host of 4 Sockets

ESXi host RAM

3 TB

Network adapters

2 x 25 Gbit as a shared network for vSAN, SAP application and admin network traffic.

See the network section of this guide for recommended network configurations.

Storage adapter

SAS 12GBS SCSI controller

Storage configuration

Disk Groups: 3

Cache Tier:  3 x 3.2 TB P4610 NVMe cache disks

Capacity Tier: 12 x 1.6 TB PM1643a SSD capacity disks

4-Socket/4-Disk Group SAP HANA Scale-Out on HCI vSAN Cluster Example Configuration

The 4-socket test environment had four vSphere ESXi host servers in the vSAN cluster. Each host had 4 Intel Optane NVMe 750G for caching layer, 3TB memory and class M based CPU. The SSD based capacity drives provided the vSAN datastore for SAP HANA scale out configuration. Up to 8 hosts (7+1 HA) can be configured for the scale out environment.

The shared partition for NFS can be provided by vSAN data store with a VM running on the HA node or a separate NFS based storage. In this config, a separate NFS based storage was used for shared partition. BWH benchmark was run and a separate ESXi host running NetWeaver VM is used as a driver to do user throughput testing.

Each vSphere ESXi server had four disk groups each consisting of 1.6TB devices for caching and four capacity tier 4TB SAS SSDs.

Two 25 Gbit network adapters were used for vSAN and other network traffic in active-standby failover manner.

Review the network section for details on how the network cards impact the number of SAP HANA VMs per host and the sizing section for how many SAP HANA VMs can be deployed on a Skylake based server system.

Each ESXi Server in the vSAN Cluster had the following configuration as shown in Table 7. This configuration supported running up to four SAP HANA VMs per HCI server.

Table 7.  SAP HANA Scale-Out Example Configuration

PROPERTY

SPECIFICATION

ESXi server model

HW Vendor SAP HANA Supported Cascade Lake  Server Model

ESXi host CPU

4 x Intel(R) Xeon(R) Gold 6230 20C CPU 2.10GHz , 4 Sockets, 20 Physical Cores/Socket

ESXi host RAM

3 TB

Network adapters

2 x 25 Gbit as a shared network for vSAN, SAP application, and admin network traffic.

See the network section of this guide for the recommended network configurations.

Storage adapter

Dell HBA330 Adapter SAS 12GBS/SATA 6GBS SCSI controller

Storage configuration

Disk Groups: 4

Cache Tier: 4 NVMe p4610 1.6TB

Capacity Tier: 16 x SAS p4510 4TB SFF

Linux LVM configuration

Linux LVM used with 8 VMDK for Data partition and 8 VMDK for Log partition

4-Socket/4-Disk Group on Cooper Lake SAP HANA HCI vSAN Based—Example 1

This 4-socket and 4-disk group test environment had a total of three vSphere ESXi servers configured and used an all-NVMe cache and capacity drives inside the hosts to provide a vSAN datastore that meets the SAP HANA HCI test requirements and KPIs.

The third node was part of the vSAN datastore and provided HA capabilities for the total 8 running SAP HANA VMs. Adding the fourth node would allow you to run more SAP HANA VMs on the vSAN datastore. The vSAN cluster limit is up to 64 cluster nodes. The vSAN cluster node number that an HCI vendor supports is determined by the vendor.

Each vSphere ESXi server had four disk groups with each consisting of one 750G Optane NVMe devices for caching and four capacity tier 1.6TB NVMe. The raw capacity of the vSAN datastore of this 3-node vSAN cluster was for data around 58 TB. The vSAN cluster limit is up to 64 cluster nodes.

Two 100 Gbit network adapters were used for vSAN and other network traffic in active-standby failover manner.

Review the network section for details about how the network cards impact the number of SAP HANA VMs per host and the sizing section for how many SAP HANA VMs can be deployed on a Skylake based server system.

Table 8.  SAP HANA Cooper Lake Example Configuration

PROPERTY

SPECIFICATION

ESXi server model

HW Vendor SAP HANA Supported Cooper Lake  Server Model

ESXi host CPU

4 x Intel(R) Xeon(R) Platinum 8376HL 28Cores CPU @ 2.6GHz,

4 Sockets, 28 Physical Cores/Socket

ESXi host RAM

1.5 TB

Network adapters

2 x 100 Gbit as a shared network for vSAN, SAP application, and admin network traffic.

See the network section of this guide for the recommended network configurations.

Storage configuration

Disk Groups: 4

Cache Tier: 4 x 750G Intel Optane SSD DC P4800x based PCIe adapters

Capacity Tier: 16 x 1.6TB Intel SSD D7-P5600 based NVMe PCIe adapters

Linux LVM configuration

Linux LVM  with 8 VMDK for Log partition

Software Resources

Table 9 shows the VMware related software resources used in this solution. From the SAP side, both SAP HANA version 1.0 and 2.0 are supported.

Table 9.  Software Resources

Software

version

purpose

VMware vCenter Server

6.7 U3 , 7.0 U1 and U2

VMware vCenter Server provides a centralized platform for managing VMware vSphere environments.

vSphere ESXi

6.7 U3, 7.0 U1 and U2

vSphere ESXi cluster to host virtual machines and provide vSAN Cluster.

VMware vSAN

6.7 U3, 7.0 U1 and U2

Software-defined storage solution for hyperconverged infrastructure

SAP HANA HCI on vSAN Configuration Guidelines

Selecting the correct components and a proper configuration is vital to achieving the performance and reliability requirements for SAP HANA. The server and vSAN configurations (RAM and CPU) determine how many SAP HANA VMs and which VMs are supported.

The vSAN configuration specifics such as the number of disk groups, selected flash devices for the caching and capacity storage tier or the vSAN network configuration define how many SAP HANA VMs a vSAN based datastore can support.

SAP HANA HCI Network Configuration

To build an SAP HANA ready HCI system, dedicated vSAN networks for SAP application, user traffic, admin and management are required. Follow the SAP HANA Network Requirements document to decide how many additional network cards have to be added to support a specific SAP HANA workload on the servers.

Table 10 provides an example configuration for an SAP HANA HCI configuration based on dual port network cards.

Table 10.                       Recommended SAP HCI on vSphere Network Configuration

 

 

ADMIN AND VMWARE HA NETWORK

APPLICATION SERVER NETWORK

vMotion  NETWORK

vSAN NETWORK

BACKUP NETWORK

SCALE-OUT INTERNODE NETWORK

SYSTEM REPLICATION NETWORK

Host Network Configuration
(all networks are dual port NICs,
configured as failover teams)

Network Label

Admin

App-Server

vMotion

vSAN

Backup

Scale-Out

HANA-Repl

Recommended MTU Size

Default (1500)

Default (1500)

9000

9000

9000

Bandwidth

1 or 10 GbE

1 or 10 GbE

>= 10 GbE*

 

>= 10 GbE*

Physical NIC

0

1

 

 

OPTIONAL

Physical NIC port

1

2

1

2

VLAN ID#

100

101

200

201

202

203

204

VM Guest Network Cards

Virtual NIC (inside VM)

0

1

-

-

2

2

3

*The selected network card bandwidth for vSAN directly influences the number of SAP HANA VMs that can be deployed on a host of the SAP HANA HCI vSAN cluster. For instance, a network exceeding 10 GE is needed if more than two SAP VMs should be deployed per host. In the stretched cluster case, the witness appliance VM can leverage the App server network. A dedicated witness network is not required.

In addition to using a high bandwidth and low latency network for vSAN, it is also strongly recommended using a VMware vSphere Distributed Switch™ for all VMware related network traffic (such as vSAN and vMotion). 

A Distributed Switch acts as a single virtual switch across all associated hosts in the data cluster. This setup allows virtual machines to maintain a consistent network configuration as they migrate across multiple hosts.

The vSphere Distributed Switch in the SAP HANA HCI solution uses at least two 10 GbE dual port network adapters for the teaming and failover purposes. A port group defines properties regarding security, traffic shaping, and NIC teaming. It is recommended to use the default port group setting except the uplink failover order that should be changed as shown in Table 10. It also shows the distributed switch port groups created for different functions and the respective active and standby uplink to balance traffic across the available uplinks.

Table 11 and Table 12 show an example of how to group the network port failover teams. It does not show the optional networks needed for VMware or SAP HANA system replication or Scale-Out internode networks. Additional network adapters are needed for these networks.

Table 11.   Minimum ESXi Server Uplink Failover Network Configuration (if all NICs have the same bandwidth to allow physical NIC failure recovery)

PROPERTY

vlan*

active uplink

standby uplink

Admin plus VMware HA Network

100

Nic0-Uplink1

Nic1-Uplink1

SAP Application Server Network

101

Nic0-Uplink2

Nic1-Uplink2

vMotion Network

200

Nic1-Uplink1

Nic0-Uplink1

vSAN Network

201

Nic1-Uplink2

Nic0-Uplink2

Table 12.        Minimum ESXi Server Uplink Failover Network Configuration (if NICs have different bandwidth, with no physical NIC failover possibility)

PROPERTY

vlan*

active uplink

standby uplink

Admin plus VMware HA Network

100

Nic0-Uplink1

Nic0-Uplink2

SAP Application Server Network

101

Nic0-Uplink2

Nic0-Uplink1

vMotion Network

200

Nic1-Uplink1

Nic1-Uplink2

vSAN Network

201

Nic1-Uplink2

Nic1-Uplink1

*VLAN id example, final VLAN numbers are determined by the network administrator.

We used different VLANs to separate the VMware operational traffic (for example: vMotion and vSAN) from the SAP and user-specific network traffic.

VMware ESXi Server: Storage Controller Mode

Server storage or RAID controller normally support both pass-through and RAID mode.

In vSAN environments, only use controllers that support pass-through mode, as it is a requirement to deactivate RAID mode and use only pass-through mode.

vSAN Disk Group Configuration

Depending on the hardware configuration, like the number of CPUs and flash devices, minimal two or more disk groups should be configured per host. More disk groups improve the storage performance. The current disk group maximum is five disk groups per vSAN cluster node.

The cache disk type should be a low-latency high-bandwidth SAS SSD or an NVMe device. The capacity should be 600 GB or larger. Review VMware vSAN design considerations for flash caching devices and the All-Flash Cache Ratio Update document for details.

In our 2 and 4-socket server reference configuration, we used two and four disk group configurations to optimally use the server hardware components installed. The number of disk groups should be selected depending on how many SAP HANA VMs will be deployed on a single vSAN cluster node. In some test configurations, we noticed that a 2-disk group configuration with NVMe for caching and four capacity disks was enough to meet the SAP HANA KPIs for two SAP HANA VMs per server. If four SAP HANA VMs should be deployed on a server, we recommend using a minimum of 3 or upto 4-disk group configuration. In addition, distribute the NVMe across all the CPUs wherever possible, if multiple NVMe are located on single CPU link, ensure that separate ports are used. As noted, the HW vendors are free to decide which configuration they want to certify.

In our Cascade Lake 2-socket/2-disk group and 4-host server vSAN cluster configuration, we used a write optimized 2x 750 GB Intel Optane DC 4800X Series for caching and 6 X 3.8TB 3840 Micron 5200 TCG-E SAS SSDs for the capacity tier per disk group. In total, we used 12 SSDs per server, all connected on one vSAN certified SCSI adapter. With this vSAN configuration, we were able to validate two SAP HANA VMs per server successfully.

In the next 4-disk group configuration, we used 4-socket in a three node vSAN cluster configuration. We used an 800GB NVMe device for caching and four SAS 900GB SSDs for the capacity tier per disk group. With this vSAN configuration, we were able to validate 4 SAP HANA VMs per server. Additional tests with other configurations also found that fewer capacity SSDs are needed to pass the tests by using NVMes in the caching tier. The number of capacity SSDs depends mainly on the size of the SAP HANA VMs that will run on the HCI solution.

Figure 9 shows a single server of our 4-disk group test configuration in a graphical way.

   A screenshot of a cell phone</p>
<p>Description automatically generated

Figure 9.                      4-disk Group Configuration Example of Our SAP HANA HCI Test Configuration

Replacing the SAS SSDs for caching with a higher bandwidth, low latency NVMe devices helped us to reduce the number of vSAN disk groups from four to two, while still maintaining the SAP HANA storage KPIs for two SAP HANA VMs per server.

Note: If 4 or more SAP HANA VMs are supported per vSAN node, it is recommended to use at least a 4-disk group configuration.

Figure 10.                  2-disk Group Configuration of Our SAP HANA HCI Test Configuration

Note: The number of capacity drives per disk group and the number of disk groups per host impact the maximum number of SAP HANA VMs per server.

vSAN Storage Configuration

All vSAN disk groups are configured as an into single cluster-wide shared datastore. The actual storage configuration for a VMDK file will be defined by a specific vSAN storage policy that fits its usage (OS, log, or data). For SAP HANA, we use the same storage policy for log and data. For the VMDK that contains the OS or SAP HANA binaries, the storage policy can be modified to use erasure coding to optimize capacity instead of performance.

vSAN Storage Policy

The SAP HANA HCI solution is a very specific configuration and is optimized for the high-performance demands of an SAP HANA database. Just like an SAP HANA appliance, the storage subsystem is optimized to meet the SAP HANA KPIs for reliability, consistency, and performance. These KPIs are validated, tested, and finally certified by SAP.

Depending on the actual vSAN hardware configuration, a vSAN storage policy is selected and applied to the SAP HANA OS, log and data disks. The vSAN policy is one of the parameters that determine the number of HANA VMs that can be supported on a server.

Table 13 and Table 14 show the vSAN storage policy we used for our testing. This storage policy should be applied on both the data disks and the log disk of each SAP HANA Database VM.

For non-performance critical volumes, the vSAN default storage policy or a custom RAID 5 storage policy can be used to lower the storage capacity need of these volumes.

For example, a VM protected by a primary level of failures to tolerate value of 1 with RAID 1 requires twice the virtual disk size, but with RAID 5, it requires 1.33 times the virtual disk size. For RAID 5, you need a minimum of 4 vSAN nodes per vSAN cluster.

The number of stripes configured should equal the number of capacity devices contained in a host. Note that the maximum number of stripes that can be configured in a storage policy is 12.

Table 13.   vSAN Storage Policy Settings for Data and Log for a Configuration with 12 or More Capacity Disks

Storage Capability

Setting

Failures to tolerance method (FTM)

RAID 1 (Mirroring)

Number of failures to tolerate (FTT)

1

Minimum hosts required

3

Number of disk stripes per object

12

Flash read cache reservation

0%

Object space reservation

100%

Disable object checksum

No

Table 14 shows the vSAN policy for a 2-disk group vSAN cluster configuration.

Table 14.   vSAN Storage Policy Settings for Data and Log for a Configuration with 8 Capacity Disks

Storage Capability

Setting

Failure Tolerance Method (FTM)

RAID 1 (Mirroring)

Number of FTT

1

Minimum hosts required

3

Number of disk stripes per object

8

Flash read cache reservation

0%

Object space reservation

100%

Disable object checksum

No

Table 15 shows the vSAN policy for less performance critical disks like OS or SAP HANA shared.

Note: You need a minimum of 4 vSAN nodes to use RAID 5, and a minimum of 6 vSAN nodes for RAID 6.

Table 15.   vSAN Storage Setting for OS or Other Less Performance Critical Volumes

Storage Capability

Setting

Failure tolerance method (FTM)

RAID 5/6 (Erasure Coding)

Number of FTT

1

Minimum hosts required

4

Number of disk stripes per object

1

Flash read cache reservation

0%

Object space reservation

0%

Disable object checksum

No

SAP HANA HCI Storage Planning and Configuration

When planning a virtualized SAP HANA environment running on vSAN, it is important to understand the storage requirements of such an environment and the applications running on it. It is necessary to define the storage capacity and IOPS needs of the planned workload.

Sizing a storage system for SAP HANA is very different from storage sizing for SAP classic applications. Unlike with SAP classic applications, where the storage subsystem is sized regarding the IOPS requirements of a specific SAP workload, the SAP HANA storage is defined by the specified SAP HANA TDI storage KPIs. These data throughput and latency KPIs are fixed values per SAP HANA instance.

Depending on how many SAP HANA VMs a vendor is supporting on an SAP HANA HCI cluster, the vSAN datastore must be able to maintain the performance KPIs and must be able to provide the storage capacity all installed SAP HANA VM on the vSAN cluster.

VMware has successfully tested a 4-socket 3-node vSAN based SAP HANA HCI configuration with an 8 SAP HANA VM configuration (4 SAP HANA VMs per one active node) and a 2-socket 4-node cluster with 2 SAP HANA VM per server configuration. These configurations represent one SAP HANA VM per socket, leaving the third or fourth node in the cluster used as an HA node. This leaves enough capacity available to support a full node failure at any time.

SAP HANA hardware configuration check tool (HWCCT) is the tool available only for SAP partners and customers that test a system and measure it against the currently valid storage KPIs. The tool and the documentation can be downloaded from SAP with a valid SAP user account. See SAP note 1943937.

SAP HANA Storage Capacity Calculation and Configuration

All SAP HANA instances have a database log, data, root, local SAP, and shared SAP volume. The storage capacity sizing calculation of these volumes is based on the overall amount of memory needed by SAP HANA’s in-memory database.

SAP has defined very strict performance KPIs that have to be met when configuring a storage subsystem. These storage performance KPIs are the leading sizing factor. How many capacity devices are used to provide the required I/O performance and latency depends on the HCI vendor hardware components.

The capacity of the overall vSAN datastore must be large enough to store the SAP HANA data, log, and binaries by providing the minimal FTT=1 storage policy. SAP has published several SAP HANA specific architecture and sizing guidelines, like the SAP HANA Storage Requirements. The essence of this guide is summarized in Figure 11 and Table 16, which can be used as a good start for planning the storage capacity needs of an SAP HANA system. It shows the typical disk layout of an SAP HANA system and the volumes needed. The volumes shown in the document should correspond with VMDK files to ensure the flexibility for the changing storage needs (the increased storage capacity most likely).

Figure 11.                  Storage Layout of an SAP HANA System. Figure © SAP SE, Modified by VMware

Table 16 determines the size of the different SAP HANA volumes. It is based on the SAP HANA Storage Requirements document. Some of the volumes like the OS and usr/sap volumes can be connected and served by one PVSCSI controller, others like the LOG and data volume are served by dedicated PVSCSI controllers to ensure high IO bandwidth and low latency.

Table 16.        Storage Layout of an SAP HANA System

Volume

VMDK

SCSI Controller

VMDK Name

SCSI ID

Sizes as of SAP HANA Storage Requirements

/(root)

Yes

PVSCSI Contr. 1

vmdk01-OS-SIDx

SCSI 0:0

Minimum 40 GiB for OS

usr/sap

Yes

PVSCSI Contr. 1

vmdk01-SAP-SIDx

SCSI 0:1

Minimum 50 GiB for SAP binaries

hana/shared

Yes

PVSCSI Contr. 1

Vmdk02-SHA-SIDx

SCSI 0:2

Minimum 1x RAM max. 1 TB

hana/data

Yes

PVSCSI Contr. 2

Vmdk03-DAT1-SIDx

Vmdk03-DAT2-SIDx vmdk03-DAT3-SIDx

SCSI 1:0

SCSI 1:1

Minimum 1 x RAM

Note: If you use multiple VMDKs, use for example Linux LVM to build one large data disk.

hana/log

Yes

PVSCSI Contr. 3

Vmdk04-LOG1-SIDx

Vmdk04-LOG2-SIDx

SCSI 2:0

SCSI 2:1

[systems <= 512GB] log volume (min) = 0.5 x RAM

[systems >= 512GB] log volume (min) = 512GB

hana/backup

(optional)

Yes

PVSCSI Contr. 4

Vmdk05-BAK-SIDx

SCSI 3:1

Size backups >= Size of HANA data+ size of redo log.

The default path for backup is (/hana/shared). Change the default value when a dedicated backup volume is used.

Figure 12 shows the default SAP HANA storage layout as documented in SAP’s HANA storage white paper, in a graphical view with the dedicated VMDKs per OS, HANA shared, log, and data volumes.

Figure 12.                  Dedicated VMDK Files per SAP HANA Volume

A “pure” VMDK solution provides the easiest way to get started and is enough to achieve the SAP HANA HCI performance and scalability KPIs.

Usage of Linux Logical Volume Manager (LVM)

Although we did not use Linux LVM with our 4-socket, 3 node cluster tests, the usage of LVM is supported and can be used if desired. Using Linux LVM can help to leverage more vSAN resources and balance the VMDK components distribution on vSAN. On the other hand, using LVM is adding configuration complexity. It is determined by the HCI solution vendor to use LVM or not.

Figure 13 shows an example how LVM can be used for the SAP HANA LOG and DATA volumes. We used this configuration with our 4-disk group SAS cache and 2-disk group configurations. Linux LVM used with 8 VMDK for data partition and a min 4 to 8 VMDK for log partition.

Note that the number of VMDK disks used for LVM should not exceed the overall number that physical capacity disks used in a host. For example, if 6 capacity disks are used in a server, 8 VMDK disks for LVM would require because two disks will have to store the objects that represent two VMDK files. This would result in an unbalanced configuration. Whereas 8 VMDK disks would be used if 8 or more capacity disks are installed. For example, if 6 capacity disks are used in a server, up to 6 VMDK disks for LVM would be ideal.

Figure 13.                  Usage of LVM Managed Volumes

vSAN Storage Capacity Calculation

To determine the overall storage capacity per SAP HANA VM, summarize the sizes of all specific and unique SAP HANA volumes as outlined in Figure 11 and Table 16.

To determine the overall vSAN datastore capacity required, without cache tier, it is required to multiply the above SAP HANA volume requirements with the amount of VMs that can run on the SAP HANA HCI cluster.

Simplified calculation:

vSAN datastore capacity = total HANA VM x (OS + USR/SAP + SHARED + DATA + LOG) x 2 (FTT=1, RAID 1) + 30% Slack space.

Example:

A cluster with 3 servers with 18 TB RAM and 8 SAP HANA VMs in total, one server is HA -> 12 TB of SAP HANA data needs to be stored.

8 x (10 GB + 60 GB + 1 TB + 12 TB / 8 VMs + 512 GB) x 2 x 1.3 = 8 x 3,142 GB x 2 x 1.3 = ~65.4 TB

We have over 138 TB of vSAN raw datastore capacity available with our example 3-node 4-socket SAP HANA HCI configuration, which is much more than the amount required for up to 8 SAP HANA VMs with a total up to 12 TB of SAP HANA system RAM capacity. The additional space can be used to store optional backups. This lowers the TCO since no additional nearline backup storage is needed. Nevertheless, moving the backup files to a permanent external backup location is recommended and required.

SAP HANA HCI VMware vSAN Detection Script

SAP requested to implement a method to detect if an SAP HANA VM runs on an SAP HANA HCI solution. VMware and SAP worked together to develop a script that allows SAP to detect at the runtime if an SAP HANA VM is running on a vSAN datastore.

The VMware vSAN detection script is available on request from VMware. Contact the author or contributors of this document for the script. The script needs to be installed by the SAP HANA HCI solution vendor on their VMware based SAP HANA HCI solution systems.

The script is a Python script that will be executed every 5 minutes on the vSAN hosts that build the SAP HANA HCI solution and will add the following information to the VM configuration file:

  • "guestinfo.vsan.enabled": True/False,
  • "guestinfo.SDS.solution": esxi_version, which is also the vSAN release version
  • "guestinfo.vm_on_vsan": True/False

This data will be parsed by the SAP CIM provider and SAP support tools like SAPsysinfo, and it detects if an SAP HANA HCI on VMware vSAN solution is used for this specific VM. Details about the VMware SAP key monitoring metrics can be found in OSS Note 2266266—Key Monitoring Metrics for SAP on VMware vSphere (version 5.5 u3 and higher).

Prerequisites:

  • Enable SSH for the time of installation. Deactivate it after the installation of the script.
  • To simplify the installation process, it is recommended to have identical password of the super user on all the ESXi servers.

Installation and execution of the health check installation script

  1. Enable SSH on the vSphere ESXi hosts systems.
  2. Place the installation script and setInfo.vib into the SAME directory of any Linux/MacOS system.
  3. Run ./install_vib_script_remotely.sh and type in the following information as prompted:
root@Centos [ ~/hana_health_check ]# ./install_vib_script_remotely.sh -h
Please specify the one of the following parameters:
-i or --install: install the vib on specified hosts
-r or --remove: remove the vib on specified hosts
-u or --update: update the vib on specified hosts
-h or --help: display this message
  1. To install the vib on the specified:
[w3-dbc302:hana_vib]$./install_vib_script_remotely.sh -i
Please specify the hosts(Separate by whitespace):
10.xx.xx.xx 
login as: root
Password:
 
Validating connectivity to 10.xx.xx.xx
10.xx.xx.xx connectivity validated
 
Uploading setInfo_v1.3.vib to 10.xx.xx.xx
Successfully uploaded setInfo_v1.3.vib to 10.xx.xx.xx:/tmp
 
Installing VIB in 10.xx.xx.xx
Host acceptance level changed to 'CommunitySupported'.
Installation Result:
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: VMware_bootbank_setInfo_1.3.0-6.7.0
   VIBs Removed:
   VIBs Skipped:
Successfully installed VIB in 10.xx.xx.xx 

Functions and scenarios of the script

The SAP HANA HCI health check Python script is executed every 5 minutes, if you want to shorten the time to 1 min, modify the value of RUN_PYC_INTERVAL at the beginning of the bash script.

For all the VMs on the host, the script gets the following info from each individual VM:

vsan.enabled = True/False
sds.solution = ESXI/vSAN_Version
vm_on_vsan = True/False

Compare those values above with the values extracted from the ESXi, then update the values on the VM if necessary.

Testing of the script

  1. Install an SAP HANA supported Linux version based VM on one of your host servers.
  2. Ensure that the latest Open VMware guest tools are installed on your test VM.
  3. Enable host and VM monitor data exchange as documented in SAP note 1606643.
  4. Download and copy the latest sapsysinfo script (<= revision 114), see SAP note 618104 for details and execute it on your test VM.
  5. Example output (search for vSAN in the document):
    1. -------------------------------------------------------------------------------
    2. VMware vSAN info:
    3.  
    4. guestinfo.vsan.enabled = 
    5. guestinfo.SDS.solution = 
    6. guestinfo.vm_on_vsan = 
    7.  
    8. -------------------------------------------------------------------------------
    9. Linux system information:
    10.  
    11. -------------------------------------------------------------------------------
  6. If you do not get the above outputs, check the prerequisites and ensure that the used hypervisor is a VMware hypervisor and the used SDS solution is vSAN.

If a VM is moved to vSAN (or have a newly created vmdk on vSAN), or moved out from vSAN to other datastores, the guestinfo will be updated within 5 mins.

If the VM is storage vMotioned to another host that does not have the script running, you need to manually delete the three variables from the VM configuration file to ensure consistency. You can check inside vCenter if these variables are set for a VM. See Figure 14 for details:

Figure 14.                  SAP HCI vSAN Specific Configuration Parameters

Updating the script

To update the vib to a higher version, replace the vib file within your Linux/MacOS directory and run the following commands:

[w3-dbc302:hana_vib]$./install_vib_script_remotely.sh -u
Please specify the hosts(Separate by whitespace):
10.xx.xx.xx
login as: root
Password:
 
Validating connectivity to 10.xx.xx.xx
10.xx.xx.xx connectivity validated
 
Uploading setInfo_v1.3.5.vib to 10.xx.xx.xx
Successfully uploaded setInfo_v1.3.5.vib to 10.xx.xx.xx:/tmp
 
Updating VIB in 10.xx.xx.xx
Installation Result:
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: VMware_bootbank_setInfo_1.3.5-6.7.0
   VIBs Removed: VMware_bootbank_setInfo_1.3.0-6.7.0
   VIBs Skipped:
Successfully update VIB in 10.xx.xx.xx

Uninstalling the script

To uninstall the script from ESXi, run:

[w3-dbc302:hana_vib]$./install_vib_script_remotely.sh -r
Please specify the hosts(Separate by whitespace):
10.xx.xx.xx
login as: root
Password:
 
Validating connectivity to 10.xx.xx.xx
10.xx.xx.xx connectivity validated
 
Removing VIB in 10.xx.xx.xx
Removal Result:
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed:
   VIBs Removed: VMware_bootbank_setInfo_1.3.5-6.7.0
   VIBs Skipped:
Successfully cleared up VMs info, removed vib from 10.xx.xx.xx

SAP HANA HCI on vSAN—Support and Process

Support Status

vSAN is supported as part of an SAP HANA HCI vendor-specific solution, as described in this document. See SAP Note 2718982.

The SAP HANA HCI solution vendor is the primary support contact for the customer with escalation possibilities to VMware and SAP. The support of an SAP HANA HCI solution is comparable and very similar to the SAP HANA Appliance support model.

The SAP HANA HCI solution vendor must define the overall usable storage capacity and performance available for SAP HANA VMs. The vendor also defines how many SAP HANA VMs per host are supported and how many vSAN cluster nodes are supported by the vendor.

For supported SAP HANA HCI solutions based on VMware products, check out SAP HANA Certified HCI Solution.

Support Process

If you want to escalate an issue with your SAP HANA HCI solution, work directly with your HCI vendor and follow the defined and agreed support process, which normally starts by opening a support ticket within the SAP support tools.

vSAN Support Insight

vSAN Support Insight is a next-generation platform for analytics of health, configuration, and performance telemetry. Its purpose is to help vSAN users maintain reliable and consistent computing, storage, and network environment and can be used, in addition to the HCI vendor specific support contract, with an active support contract running vSphere 6.5 U1d or later.

How does it work?

See How your Experiences Improve Products and Services for details.

Conclusion

With vSAN, VMware equips its hardware and OEM partners to build SAP HANA ready HCI solutions. Not only compute, memory, and network, but also internal storage components can be leveraged from a standard x86 server, which dramatically changes the SAP HANA infrastructure economics and the deployment methodology. With SAP HANA HCI powered by VMware solutions, an SAP hardware partner vendor can build true turnkey SAP HANA infrastructure solutions offering customers to deploy SAP HANA in days instead of weeks or months.

 

Appendix

Optimizing SAP HANA on vSphere 6.5 and later - Configuration Parameter List

VMware vSphere can run a single large or multiple smaller SAP HANA virtual machines on a single physical host. This section describes how to optimally configure a VMware virtualized SAP HANA environment. These parameters are valid for SAP HANA VMs running in a standard vSphere environment or SAP HANA on vSAN based SAP HANA HCI configurations and are valid for 2-, 4- and 8-socket hosts.

The listed parameters settings are the recommended settings for the physical server, the vSphere ESXi host, the VM, and the Linux OS to achieve optimal operational readiness and stable performance for SAP HANA on vSphere. These parameter settings are the default settings that should be configured for SAP HANA on vSphere.

The settings described in the section “low latency settings” should only be applied in rare situations where SAP HANA must perform with the lowest latency possible.

The shown parameters are the best practice configuration parameters and in case of an escalation, the VMware or HCI vendor SAP support engineers will verify these settings.

Physical HOST BIOS Parameter Settings

DESCRIPTION

UEFI BIOS HOST

It is recommended to use (U)EFI BIOS as the standard BIOS version for physical vSphere hosts. All SAP HANA appliance server configurations leverage (U)EFI as the standard BIOS. vSphere fully supports EFI since version 5.0.

Enable Intel VT technology

Enable all BIOS virtualization technology settings.

Configure RAM hemisphere mode

Distribute DIMM modules in a way to achieve best performance (hemisphere mode), use the fastest DIMM modules available for the selected RAM size. Beware of the CPU specific optimal RAM configurations that depend on the available memory channels per CPU.

CPU - Populate all available CPU sockets, use a fully meshed QPI NUMA architecture

To avoid timer synchronization issues, use a multi-socket server that ensures NUMA node timer synchronization. NUMA systems that do not run synchronization will need to synchronize the timers in the hypervisor layer, which can impact performance. See Timekeeping in VMware Virtual Machines for reference.

Select only SAP HANA vSphere supported CPUs. Verify the support status with SAP note 2652670—SAP HANA VM on VMware vSphere.

Enable CPU Intel Turbo Boost

Allow Intel automatic CPU core overclocking technology (P-States).

Disable QPI Power Management

Static high power for QPI links.

Enable hyperthreading

Double the logical CPU cores to allow ESXi to take advantage for the more available CPU threads. Enable HT regardless the Intel L1 Terminal Fault issue, but make sure to install the latest VMware vSphere patches for this issue.

Enable execute disable feature

Enable the Data Execution Prevention bit (NX-bit), required for vMotion.

Disable node interleaving

Disable node interleaving in BIOS.

Disable C1E Halt state

Disable Enhanced C-States in BIOS.

Set Power Management to High Performance

Do not use any power management features on the server like C-States. Configure in the BIOS static high performance

Set correct PMem mode as specified by the HW vendor for either app-direct or memory mode

Follow the vendor documentation and enable PMem for the usage with ESXi. Note, only app direct memory mode is supported with production level VMs.

 

Memory mode is only supported with a ratio of 1:4. SAP provides as of today only no-production workload support.

 

VMware vSAN does not have support for “App-Direct” mode (source) as cache or as a capacity tier device of vSAN. However, vSAN will work with vSphere hosts equipped with Intel Optane PMem in App-Direct mode and SAP HANA VM’s can leverage PMEN according to SAP Note 2913410. Please note that the VMware HA restriction (as described in note 2913410) needs to be considered.

Disable all unused BIOS features, such as:

Video BIOS Shadowable, Video RAM Cacheable, on-board audio, on-board modem, on-board serial ports, on-board parallel ports, on-board game port, floppy drive, CD-ROM, USB.

 

VSPHERE ESXi HOST Parameter Settings

DESCRIPTION

Networking

Use: Virtual Distributed Switches (vDS) to connect all hosts that work together. Define the port groups that are dedicated to SAP HANA, management, and vMotion traffic. Use at least the dedicated 10GbE for vMotion and the SAP App server or replication networks. At least 25 GbE for vMotion for SAP HANA system >= 2 TB is recommended.

Storage

Configuration

If you want to use vSAN, select one of the certified SAP HANA vSAN based HCI solutions and refer to the storage section of this document.

 

When creating your storage disks for SAP HANA on the VM/OS level, ensure that you can maintain the SAP specified TDI storage KPIs for data and log. Use the storage layout and LVM configuration as a template.

SAP Monitoring

Enable SAP monitoring on the host -> Misc.GuestLibAllowHostInfo = “1” Details: SAP note 1409604.

 

Without this parameter, no host performance relevant data will be viewable inside an SAP monitoring enabled VM.

 

SAP HANA VIRTUAL MACHINE Parameter Settings

DESCRIPTION

Tips how to edit the .vmx file

Review tips for editing a .vmx file in VMware KB 1714.

UEFI BIOS Guest

It is recommended to use (U)EFI BIOS as the standard BIOS version for vSphere hosts and guests. Only with EFI features like Secure Boot are possible. See VMware DOC-28494 for details.  You can configure this with the vSphere client by choosing EFI boot mode.

If you are using vSphere 6.0 and you only see 2 TB memory in the guest, upgrade to the latest ESXi 6.0 version.

SAP monitoring

Enable SAP monitoring inside the SAP (HANA) VM with the advanced VM configuration parameter tools.guestlib.enableHostInfo = "TRUE".

Details: SAP note 1409604.

Beside setting this parameter, the VMware guest tools need to be installed. See VMware KB 1014294 for details.

vCPU hotplug

Ensure that the vCPU hotplug is deactivated; otherwise, vNUMA is disabled and SAP HANA will have a negative performance impact. Details: VMware KB 2040375.

Memory reservations

Set fix memory reservations for SAP HANA VMs. Do not overcommit memory resources.

 

It is required to reserve memory for the ESXi host. It is recommended to reserve, depending on the amount of CPU sockets memory for ESXi.

 

Typical memory reservations for a host is between 32-64 GB for a 2-socket server, 64-128 GB for a 4-socket server and 128-256 GB for an 8-socket server. These are not absolute figures, since the memory need of ESXi depends strongly on the actual HW, ESXi, and VM configuration and enabled ESXi features like vSAN.

CPU

Do not overcommit CPU resources and configure the dedicated CPU resources per SAP HANA VM.

 

You can use hyperthreads when you configure the virtual machine to gain the additional performance. For CPU generations earlier than Cascade Lake, you should consider disabling hyperthreading due to the Intel Vulnerability Foreshadow L1 Terminal Fault. See KB55636 for details.

 

You must configure 2 x the cores per CPU socket of a VM for using hypterthreads. For example, 2-socket wide VM on a 28 core Cascade Lake system will require 112 vCPUs.

vNUMA nodes

SAP HANA on vSphere can be configured to leverage half-CPU and full-CPU sockets. A half-CPU socket is configured by only half of the available physical cores of a CPU socket in the VM configuration GUI.

 

The vNUMA nodes of the VM will always be >=1, depending on how many CPUs you have configured in total.

 

If you need to access an additional NUMA node, use all CPU cores of this additional NUMA node. Nevertheless, use as few NUMA nodes as possible to optimize memory access.

Align virtual CPU VM configuration to actual server HW

Example: A “half-socket” VM running on a server with 28-core CPUs, should be configured with 28 virtual CPUs to leverage 14 cores and 14 hyperthreads per CPU socket. A full socket VM should also be configured to use 56 vCPUs to leverage all 28 physical CPU cores and available hyperthreads per socket.

Paravirtualized SCSI driver for I/O devices

Use the dedicated SCSI controllers for OS, log, and data, to separate disk I/O streams. For details, see the SAP HANA disk layout section.

Use Virtual Machine File System

Use VMDK disks whenever possible to allow optimal operation via the vSphere stack. In-guest NFS mounted volumes for SAP HANA are supported as well.

Create datastores for SAP HANA data and log files

Ensure the storage configuration passes the SAP defined storage KPIs for TDI storage. Use the SAP HANA hardware configuration check tool (HWCCT) to verify your storage configuration. See SAP note 1943937 for details.

We recommend eagerzeroed thick virtual disks for data and log disk

This avoids lazy zeroing (initial write penalty).

VMXNET3

Use paravirtual VMXNET 3 virtual NICs for SAP HANA virtual machines.

 

We recommend at least 3-4 different NICs inside a VM in the HANA VM (App/Management Server Network, Backup Network, and if needed HANA System Replication Network). Corresponding physical NICs inside the host are required.

Optimize the application server network latency if required

Disable virtual interrupt coalescing for VMXNET 3 virtual NICs that communicate with the App servers or front end to optimize network latency. Do not set this parameter for throughput-oriented networks like vMotion or SAP HANA system replication. Use the advanced options in the vSphere Web Client or directly modify the .vmx file and add ethernetX.coalescingScheme = “disable”. X stands for your network card number. See Best Practice for Performance Tuning of Latency-Sensitive Workloads in vSphere VMs for detail.

Set lat.Sensitivity = normal

Check with the vSphere client and ensure that in the VM configuration, the value of Latency Sensitivity Settings is set to "normal”. Restart the VM to change this setting.

Do not change this setting to “high” or “low”. Change this setting under the instruction of VMware support engineers.

Associate Virtual Machines with Specified NUMA Nodes to optimize NUMA memory locality

Associating a NUMA node with a virtual machine to specify the NUMA node affinity, is constraining the set of NUMA nodes on which NUMA can schedule a virtual machine's virtual CPU and memory.

 

Use the vSphere Web Client or directly modify the .vmx file and add numa.nodeAffinity=x (for example: 0,1).

 

Note: This is only needed if the VM is < the available CPU sockets of the ESXi host. For half-socket SAP HANA VM configurations, use the VMX parameter sched.vCPUXx.affinity as document in the next section.

 

Procedure:

  1. Browse to the cluster in the vSphere Client. 
  2. Click the Configure tab and click Settings. 
  3. Under VM Options, click the Edit button. 
  4. Select the VM Options tab and expand Advanced
  5. Under the Configuration Parameters, click the Edit Configuration button. 
  6. Click Add Row to add a new option. 
  7. In the Name column, enter numa.nodeAffinity. 
  8. In the Value column, enter the NUMA nodes where the virtual machine can be scheduled. 

Use a comma-separated list for multiple nodes. For example, enter 0,1 to constrain the virtual machine resource scheduling to NUMA nodes 0 and 1. 

  1. Click OK
  2. Click OK to close the Edit VM dialog box.

 

Attention:

When you constrain the NUMA node affinities, you might interfere with the ability of the ESXi NUMA scheduler to rebalance virtual machines across NUMA nodes for fairness. Specify the NUMA node affinity only after you consider the rebalancing issues.

See Associate Virtual Machines with Specified NUMA Nodes for details.

Configure virtual machines to use hyperthreading with NUMA

 

 

For memory latency sensitive workloads with low processor utilization, such as SAP HANA, or high interthread communication, we recommended using hyperthreading with fewer NUMA nodes instead of full physical cores spread over multiple NUMA nodes.  Use hyperthreading and enforce NUMA node locality per KB 2003582.

 

This parameter is only required when hyperthreading should be leveraged for a VM. Using hyperthreading can increase the compute throughput but may increase the latency of threads.

Note: This parameter is only important for half-socket and multi-VM configurations that do not consume the full server, like a 3-socket VM on a 4-socket server. Do not use it when a VM leverages all installed CPU sockets, e. g. 4-socket wide VM on a 4-socket host or 8-socket VM on an 8-socker host.

 

Use the vSphere Web Client and add the following advanced VM parameter:

numa.vcpu.preferHT=”TRUE” (per VM setting) or as a global setting on the host:  Numa.PreferHT=”1” (host).

 

Note: For non-mitigated CPUs like Haswell, Broadwell and Skylake you may consider not to use hyperthreading at all. Check KB 55806 for details.

 

PMem enabled VMs

To configure an SAP HANA Optane PMem-enabled VM for optimal performance, it is necessary to align the VM configuration to the underlying HW and especially NUMA configuration.

 

VMware KB 78094 provides information about how to configure the NVDIMMs (VMware’s representation of Optane PMem) correctly and align the NVDIMMs to the physical NUMA architecture of the physical server.

 

By default, Optane PMem allocation in vmkernel for VM NVDIMMs does not consider NUMA. This can result in the VM running on a certain NUMA node and Optane PMem allocated from a different NUMA node. This will cause NVDIMMs access in the VM to be remote, resulting in poor performance. To solve this, you must add the following settings to a VM configuration using vCenter:

 

Example for a 4-socket wide VM:

  • nvdimm0:0.nodeAffinity=0
  • nvdimm0:1.nodeAffinity=1
  • nvdimm0:2.nodeAffinity=2
  • nvdimm0:3.nodeAffinity=3

 

Parameter sched.pmem.prealloc=TRUE is an optional parameter equivalent to eager zero thick provisioning of VMDKs and improves initial writes to Optane PMem.

 

Also configure the CPU NUMA node affinity or CPU affinities.

Remove unused devices

Remove floppy disks or CD-ROM to release resources and to mitigate possible errors.

 

LINUX OS Parameter Settings

DESCRIPTION

Linux kernel

VMware strongly recommends using the latest SAP HANA supported kernel version

Configure C-States and BIOS EPB for lower latency in Linux

Disable in BIOS (for example, static high performance or OS controlled) If disabled in BIOS, optional: intel_idle.max_cstate=0

If disabled in BIOS, optional: processor.max_cstate=0

Set Energy Performance in BIOS to “maximum performance”, optional: cpupower set -b 0

Optional: Disable Large Receive Offload (LRO) in the Linux guest operating system to lower latency for client/application server-facing NIC adapter

This helps to lower network latency of client/application server facing NIC adapters run: “ethtool -K ethY lro off”.

Do not disable LRO for throughput NIC adapters such as for backup, replication or SAP HANA internode communication networks.

Works only with Linux kernel 2.6.24 and later and uses a VMXNET3.

Additional details: http://kb.vmware.com/kb/2055140

SLES Linux page cache limits

Follow SAP Note 1557506 and VMware vSphere configuration guidelines, SAP note 2161991 to use the optimal configuration settings.

Disable I/O scheduling

SLES15SP2 onwards this is configured by default,  scheduler is set to none with block_mq enabled.

To optimize large-scale workloads with intensive I/O patterns, change the queue depths of the SCSI default values

The large-scale workloads with intensive I/O patterns require adapter queue depths greater than the Paravirtual SCSI (PVSCSI) default values. The default values of PVSCSI queue depth are 64 (for device) and 254 (for adapter). You can increase PVSCSI queue depths to 254 (for device) and 1024 (for adapter) inside a Windows virtual machine or Linux virtual machine.

Create a file of any name in the /etc/modprobe.d/ directory with this line:

options vmw_pvscsi cmd_per_lun=254 ring_pages=32

Note: For RHEL5, edit /etc/modprobe.conf with the same line. Make a new initrd for the settings to take effect. You can do this either by using mkinitrd, or by re-running vmware-config-tools.pl.

 

Starting in version 6, RHEL uses modprobe.d.

Alternatively, append these to kernel boot arguments (for example, on Red Hat Enterprise Linux edit /etc/grub.conf or on Ubuntu edit /boot/grub/grub.cfg).

    vmw_pvscsi.cmd_per_lun=254

    vmw_pvscsi.ring_pages=32

Reboot the virtual machine. See VMware KB 2053145 for details.

Note: Review the VMware KB article 2088157 to ensure that the minimum VMware patch level is used to avoid possible virtual machine freezes under heavy I/O load.

Install the latest version of VMware Tools

VMware Tools is a suite of utilities, which enhances the performance of the virtual machine’s guest operating system and improves the vm management. See http://kb .vmware com/kb/1014294 for details.

Configure NTP time server

Use the same external NTP server as configured for vSphere. For details: SAP note 989963.

 

General SAP HANA Linux Configuration Recommendations:

 

Linux Operating System with SAP HANA Reference Guide

See the recommended operating system configuration settings for running SAP HANA on Linux in the Reference Guide.

Disable AutoNUMA

Later Linux kernel (RHEL 7 and SLES 12) supporting auto- migration according to NUMA statistics.

For SLES: # yast bootloader, choose “Kernel Parameters” tab (ALT-k) and edit the “Optional Commandline Parameters” section by appending numa_balancing=disabled

For RHEL add “kernel.numa_balancing = 0” to /etc/sysctl .d/sap_hana.conf and reconfigure the kernel by running: # sysctl -p /etc/sysctl.d/sap_hana.conf

Use Block mq

In the kernel parameters add scsi_mod.use_blk_mq=1.

For OS version Sles15 SP2 and beyond this is enabled by default.

Disable transparent HugePages

THP is not supported for the use with SAP HANA DB, as it may lead to hanging situations and performance degradations.

To check the current configuration, run the following command: # cat/sys/kernel/mm/transparent_hugepage/enabled

Its output should read: always madvise [never]

If this is not the case, you can disable the THP usage at runtime by issuing the following command: # echo never > /sys/kernel/mm/ transparent_hugepage/enabled

For details, refer to the SAP WIKI for SAPs and the Linux OS vendors virtualization independent recommended Linux OS settings for SAP HANA.

Change the following parameters in /etc/sysctl.conf

 

(important for SAP HANA Scale-Out deployments)

net.core.rmem_default = 262144

net.core.wmem_max = 8388608

net.core.wmem_default = 262144

net.core.rmem_max = 8388608

net.ipv4.tcp_rmem = 4096 87380 8388608

net.ipv4.tcp_wmem = 4096 65536 8388608

net.ipv4.tcp_mem = 8388608 8388608 8388608

net.ipv4.tcp_slow_start_after_idle = 0

Example Linux kernel boot loader parameters

intel_idle.max_cstate=0 processor.max_cstate=0 numa_balancing=disabled transparent_hugepage=never elevator=noop vmw_pvscsi.cmd_per_lun=254 vmw_pvscsi.ring_pages=32

Performance Optimization for Low Latency SAP HANA VMs

Further optimization of virtual SAP HANA performance can be required when SAP HANA must perform as close to bare-metal as possible and with the shortest latency in terms of database access times.

When optimizing SAP HANA for low latency, we recommend sizing an SAP HANA VM with the least number of NUMA nodes. When an SAP HANA VM needs more CPU or RAM than a single NUMA node provides, configure an additional NUMA node and its resources.

To achieve the optimal performance for an SAP HANA virtual machine, use the settings as described in the next table in addition to the previously described settings. In terms of CPU scheduling and priority, these settings improve performance by reducing the amount of vCPU and vNUMA migration, while increasing the priority of the SAP HANA production virtual machine.

CPU Affinities

By specifying a CPU affinity setting for each virtual machine, you can restrict the assignment of virtual machines to a subset of the available processors (CPU cores) in multiprocessor systems. By using this feature, you can assign each virtual machine to processors in the specified affinity set.

Setting CPU affinities can decrease the CPU and memory latency by not allowing the ESXi scheduler to migrate VM threads to other logical processors. Setting CPU affinities is required when configuring so called SAP HANA half-socket VMs.

Before you use CPU affinity, you need to take the following items into consideration:

  • For multiprocessor systems, ESXi systems perform automatic load balancing. Avoid the manual specification of virtual machine affinity to improve the scheduler’s ability to balance load across processors.
  • Affinity can interfere with the ESXi host’s ability to meet the reservation and shares specified for a virtual machine.
  • Because CPU admission control does not consider affinity, a virtual machine with manual affinity settings might not always receive its full reservation. Virtual machines that do not have manual affinity settings are not adversely affected by virtual machines with manual affinity settings.
  • When you move a virtual machine from one host to another, affinity might no longer apply because the new host might have a different number of processors.
  • The NUMA scheduler might not be able to manage virtual machine that is already assigned to the certain processors using affinity.
  • Affinity can affect the host’s ability to schedule virtual machines on multicore or hyperthreaded processors to take full advantage of resources shared on such processors.

For more information about performance practices, see the vSphere Resource Management Guide, also refer to the following documentation around specifying NUMA controls.

Additional Performance Tuning Settings for SAP HANA Workloads

Note: The following parameters are optional parameters and are only needed for the lowest CPU latency. Set these parameters with caution.

SAP HANA VIRTUAL MACHINE Parameter Settings

DESCRIPTION

Tips about how to edit the .vmx file

Review tips for editing a .vmx file in VMware KB 1714

monitor.idleLoopSpinBeforeHalt = “true”

 

and monitor.idleLoopMinSpinUS = “xx us”

 

 

Setting these advanced VM parameters can help to improve performance of a VM at the cost of CPU time on the ESXi host and should only get configured for SAP HANA workload that runs anyhow as the only workload on a NUMA node/complete server.

 

Edit the vmx file and add the following two advanced parameters: monitor.idleLoopSpinBeforeHalt = “true” AND

monitor.idleLoopMinSpinUS = “xx” (For example: 50)

Both parameters must be configured to influence the de-scheduling time.

 

Background: The guest operating system issues a Halt instruction which stops (or de-schedules) the vCPU on the ESXi host. Keeping the virtual machine spinning longer before Halt negates the number of inter-processors wake up requests.

Set control.halt_in_monitor = “TRUE”

 

 

 

In the default configuration of ESX 7.0, idle state of guest HLT instruction will be emulated w/o leaving VMM if a vCPU has exclusive affinity. If affinity is non-exclusive, guest HLT will be emulated in vmkernel, which may result in having vCPU de-scheduled from the physical CPU and can lead to longer latencies; therefore, it is recommended to set this parameter to “TRUE”, to ensure that the HLT instruction gets emulated inside the VMM and not in the vmkernel.

 

Use the vSphere Web Client and add the following advanced VM parameter: monitor_control.halt_in_monitor = “TRUE”.

Set monitor_control.disable_pause_loop_exiting = “TRUE”

This parameter prevents the VM from exiting to the hypervisor unnecessarily during a pause instruction. This is specific to Intel Skylake or later CPU based systems.

Configuring CPU Affinity

 

sched.vcpuXx.affinity = “Yy-Zz”

 

 

 

Note: Remove the numa.nodeAffinity settings if set and if the CPU affinities with sched.vCPUxxx.affiity are used.

By specifying a CPU affinity setting for each virtual machine, you can restrict the assignment of virtual machines to a subset of the available processors (CPU cores) in multiprocessor systems. By using this feature, you can assign each virtual machine to processors in the specified affinity set.

See Scheduler operation when using the CPU Affinity (2145719) for details.

This is especially required when configuring so-called SAP HANA “half-socket” VM’s or for very latency critical SAP HANA VMs. It is also required when parameter numa.slit.enable gets used.

Just like with numa.NodeAffinity it is possible to decrease the CPU and memory latency by further limit the ESXi scheduler to migrate VM threads to other logical processors / CPU threads by leveraging the sched.vCPUxxx.affinity VMX parameter in contrast to parameter numa.NodeAffinity it is possible assign a vCPU to a specify physical CPU thread and is for instance necessary for instance when configuring half-socket SAP HANA VMs.

Use the vSphere Web Client or directly modify the .vmx file (recommended way) and add sched.vcpuXx.affinity = "Yy-Zz" (for example: sched.vcpu0.affinity = "0-55") for each virtual CPU you want to use.

 

Procedure:

  1. Browse to the cluster in the vSphere Client. 
  2. Click the Configure tab and click Settings. 
  3. Under VM Options, click the Edit button. 
  4. Select the VM Options tab and expand Advanced. 
  5. Under Configuration Parameters, click the Edit Configuration button. 
  6. Click Add Row to add a new option. 
  7. In the Name column, enter sched.vcpuXx.affinity (Xx stands for the actual vCPU want to assign to a physical CPU thread). 
  8. In the Value column, enter the physical CPU threads where the vCPU can be scheduled. E.g. enter 0-55 to constrain the virtual machine resource scheduling to physical CPU threads 0-55, which would be the 1st CPU of an 28 core CPU host. 
  9. Click OK. 
  10. Click OK to close the Edit VM dialog box.

 

For more information about potential performance practices, see vSphere Resource Management Guide.

>4-socket VM on 8-socket hosts

Add the advanced parameter: numa.slit.enable = "TRUE" to ensure the correct NUMA map for VMs > 4 socket on 8-socket hosts.

Note: sched.vcpuXx.affinity = "Yy-Zz" must get configured when numa.slit.enable is set to "TRUE"

 

 

Example VMX Configurations for SAP HANA VMs

The following examples provide an overview about how to set the additional VMX parameters for SAP HANA half- and full-CPU socket VMs. These parameters should be added in addition to the standard parameters via the vSphere Web Client or by directly adding these parameters to the .vmx file with a text editor.

SAP HANA Half-Socket VM Additional VMX Parameters

Settings

1st-“half-socket” VM on socket 0 of a 28-core CPU server:

  • numa.vcpu.preferHT=”TRUE”
  • sched.vcpu0.affinity = "0-27"
  • sched.vcpu1.affinity = "0-27"
  • sched.vcpu2.affinity = "0-27"
  • sched.vcpu26.affinity = "0-27"
  • sched.vcpu27.affinity = "0-27"

 

2nd-“half-socket” VM on socket 0 of a 28-core CPU server:

  • numa.vcpu.preferHT=”TRUE”
  • sched.vcpu0.affinity = "28-55"
  • sched.vcpu1.affinity = "28-55"
  • sched.vcpu2.affinity = "28-55"
  • sched.vcpu26.affinity = "28-55"
  • sched.vcpu27.affinity = "28-55"

 

     

 

SAP HANA Half-Socket PMem VM Additional VMX Parameters

Settings

1st-“half-socket” VM on socket 1 of a 28-core CPU server:

  • nvdimm0:1.nodeAffinity=1
  • numa.vcpu.preferHT=”TRUE”
  • sched.vcpu0.affinity = "56-83"
  • sched.vcpu1.affinity = "56-83"
  • sched.vcpu2.affinity = "56-83"
  • sched.vcpu26.affinity = "56-83"
  • sched.vcpu27.affinity = "56-83"

 

2nd-“half-socket” VM on socket 1 of a 28-core CPU server:

  • nvdimm0:1.nodeAffinity=1
  • numa.vcpu.preferHT=”TRUE”
  • sched.vcpu0.affinity = "84-111"
  • sched.vcpu1.affinity = "84-111"
  • sched.vcpu2.affinity = "84-111"
  • sched.vcpu26.affinity = "84-111"
  • sched.vcpu27.affinity = "84-111"

 

 

SAP HANA 1-Socket VM Additional VMX Parameters

Settings

1-Socket VM on socket 3 of a 28-core CPU server:

  • numa.vcpu.preferHT=”TRUE”
  • numa.nodeAffinity=1

1-Socket PMEM VM on socket 3 of a 28-core CPU server:

  • nvdimm0:1.nodeAffinity=3
  • numa.vcpu.preferHT=”TRUE”
  • numa.nodeAffinity=3

 

SAP HANA 2-Socket VM Additional VMX Parameters

Settings

2-Socket VM on socket 0 and 1 of a 28-core CPU server:

  • numa.vcpu.preferHT=”TRUE”
  • numa.nodeAffinity=0,1

2-Socket PMEM VM on socket 0 and 1 of a 28-core CPU server:

  • nvdimm0:0.nodeAffinity=0
  • nvdimm0:1.nodeAffinity=1
  • numa.vcpu.preferHT=”TRUE”
  • numa.nodeAffinity=0,1

 

SAP HANA 3-Socket VM Additional VMX Parameters

Settings

3-Socket VM on socket 0 and 1 of a 28-core CPU server:

  • numa.vcpu.preferHT=”TRUE”
  • numa.nodeAffinity=0,1,2

3-Socket PMEM VM on socket 0 and 1 of a 28-core CPU server:

  • nvdimm0:0.nodeAffinity=0
  • nvdimm0:1.nodeAffinity=1
  • nvdimm0:2.nodeAffinity=2
  • numa.vcpu.preferHT=”TRUE”
  • numa.nodeAffinity=0,1,2

 

SAP HANA 4-Socket VM Additional VMX Parameters

Settings

4-Socket VM on a 4-socket a 28-core CPU server:

  • no additional settings are required since the VM utilizes all server resources!

4-Socket PMEM VM a 4-socket a 28-core CPU server:

  • nvdimm0:0.nodeAffinity=0
  • nvdimm0:1.nodeAffinity=1
  • nvdimm0:2.nodeAffinity=2
  • nvdimm0:3.nodeAffinity=3

 

SAP HANA 5-Socket VM Additional VMX Parameters

Settings

5-Socket VM on a 8-socket a 28-core CPU server running on socket 0-4:

  • numa.slit.enable = "TRUE"
  • sched.vcpu0.affinity = "0-55"
  • sched.vcpu1.affinity = "0-55"
  • sched.vcpu2.affinity = "0-55"
  • sched.vcpu222.affinity = "168-223"
  • sched.vcpu223.affinity = "168-223"

5-Socket PMEM VM a 8-socket a 28-core CPU server running on socket 0-4:

  • nvdimm0:0.nodeAffinity=0
  • nvdimm0:1.nodeAffinity=1
  • nvdimm0:2.nodeAffinity=2
  • nvdimm0:3.nodeAffinity=3
  • nvdimm0:2.nodeAffinity=4
  • numa.slit.enable = "TRUE"
  • sched.vcpu0.affinity = "0-55"
  • sched.vcpu1.affinity = "0-55"
  • sched.vcpu2.affinity = "0-55"
  • sched.vcpu222.affinity = "168-223"
  • sched.vcpu223.affinity = "168-223"

 

SAP HANA 8-Socket VM Additional VMX Parameters

Settings

8-Socket VM on a 8-socket a 28-core CPU server:

  • numa.slit.enable = "TRUE"
  • sched.vcpu0.affinity = "0-55"
  • sched.vcpu1.affinity = "0-55"
  • sched.vcpu2.affinity = "0-55"
  • sched.vcpu222.affinity = "168-223"
  • sched.vcpu223.affinity = "168-223"

8-Socket PMEM VM a 8-socket a 28-core CPU server:

  • nvdimm0:0.nodeAffinity=0
  • nvdimm0:1.nodeAffinity=1
  • nvdimm0:2.nodeAffinity=2
  • nvdimm0:3.nodeAffinity=3
  • nvdimm0:4.nodeAffinity=4
  • nvdimm0:5.nodeAffinity=5
  • nvdimm0:6.nodeAffinity=6
  • nvdimm0:7.nodeAffinity=7
  • numa.slit.enable = "TRUE"
  • sched.vcpu0.affinity = "0-55"
  • sched.vcpu1.affinity = "0-55"
  • sched.vcpu2.affinity = "0-55"
  • sched.vcpu222.affinity = "168-223"
  • sched.vcpu223.affinity = "168-223"

 

SAP HANA Low-latency VM Additional VMX Parameters

Settings

4-Socket Low Latency VM on a 28-core CPU server:

  • monitor.idleLoopSpinBeforeHalt = "TRUE"
  • monitor.idleLoopMinSpinUS = "50"
  • monitor_control.disable_pause_loop_exiting = "TRUE" (when Skylake or later)
  • sched.vcpu0.affinity = "0-55"
  • sched.vcpu1.affinity = "0-55"
  • sched.vcpu2.affinity = "0-55"
  • sched.vcpu222.affinity = "168-223"
  • sched.vcpu223.affinity = "168-223"

 

4-Socket Low Latency PMem VM on a 28-core CPU server:

  • monitor.idleLoopSpinBeforeHalt = "TRUE"
  • monitor.idleLoopMinSpinUS = "50"
  • monitor_control.disable_pause_loop_exiting = "TRUE"
  • nvdimm0:0.nodeAffinity=0
  • nvdimm0:1.nodeAffinity=1
  • nvdimm0:2.nodeAffinity=2
  • nvdimm0:3.nodeAffinity=3
  • sched.vcpu0.affinity = "0-55"
  • sched.vcpu1.affinity = "0-55"
  • sched.vcpu2.affinity = "0-55"
  • sched.vcpu222.affinity = "168-223"
  • sched.vcpu223.affinity = "168-223"

 

     

 

SAP HANA Low-latency VM Additional VMX Parameters

Settings

8-Socket Low Latency VM on a 28-core CPU 8-socket server:

  • numa.slit.enable = "TRUE" (>4 socket VM)
  • monitor.idleLoopSpinBeforeHalt = "TRUE"
  • monitor.idleLoopMinSpinUS = "50"
  • monitor_control.disable_pause_loop_exiting = "TRUE"
  • sched.vcpu0.affinity = "0-55"
  • sched.vcpu1.affinity = "0-55"
  • sched.vcpu2.affinity = "0-55"
  • sched.vcpu222.affinity = "392-447"
  • sched.vcpu223.affinity = "392-447"

 

8-Socket Low Latency PMem VM on a 28-core CPU 8-socket server:

  • numa.slit.enable = "TRUE" (>4 socket VM)
  • monitor.idleLoopSpinBeforeHalt = "TRUE"
  • monitor.idleLoopMinSpinUS = "50"
  • monitor_control.disable_pause_loop_exiting = "TRUE"
  • nvdimm0:0.nodeAffinity=0
  • nvdimm0:1.nodeAffinity=1
  • nvdimm0:2.nodeAffinity=2
  • nvdimm0:3.nodeAffinity=3
  • nvdimm0:4.nodeAffinity=4
  • nvdimm0:5.nodeAffinity=5
  • nvdimm0:6.nodeAffinity=6
  • nvdimm0:7.nodeAffinity=7
  • sched.vcpu0.affinity = "0-55"
  • sched.vcpu1.affinity = "0-55"
  • sched.vcpu2.affinity = "0-55"
  • sched.vcpu222.affinity = "392-447"
  • sched.vcpu223.affinity = "392-447"

 

     

 

CPU Thread Matrix

The following table shows the CPU thread matrix of a 28 core CPU as a reference when configuring the sched.vCPUXx.affinty =”Xx-Yy” parameter. The list shows the staring and end range required for the “Xx-Yy” parameter. For example, for CPU 5 it would be “280-335”.

 

Monitoring and Verifying an SAP HANA Installation

  • SAP Note 1698281 provides information about how you can monitor the data growth and the utilization of actual memory. With this, it is also possible to detect and diagnose the memory leaks during operation.
  • SAP Note 1969700 covers all the major HANA configuration checks and presents a tabular output with configurations that are changed. The collection of SQL statements is very helpful in checking and identifying parameters that are configured and conflict with the SAP recommended configuration parameters.

Revision History

We included the following updates for vSphere 7.0 U2:

Author and Contributors

  • Erik Rieger, Principal SAP Global Technical Alliance Manager and Architect, VMware Global SAP Alliance wrote the original version of this paper and helped updating it with Scale-Out and vSphere 7.0 U1/U2 content.
  • Sathya Krishnaswamy, VMware Staff Performance Outbound Engineering team, modified for Cascade Lake scale up for 7.0 U1/U2 and scale out configurations on this paper.
  • Chen Wei, Staff Solutions Architect in the Solutions Architecture team of the Cloud Platform Business Unit contributed to the original version of this paper.
  • Todd Muirhead, VMware Staff Performance of the VMware Performance Unit contributed to the original version of this paper.
  • Sebastian Lenz, VMware SAP Performance and Certification Engineer of the VMware Performance Unit contributed to the original version of this paper.

 

 


[1] vSAN 7.0 is not supported.

[2] This is a support rather than technical limitation.

[*] vSAN 7.0 U1/U2 File Service (NFS) is not supported. **Stretched Cluster Documentation

 

Filter Tags

vSAN Reference Architecture