]

Category

  • Reference Architecture

Product

  • vSAN

SAP HANA on Hyperconverged Infrastructure (HCI) Solutions Powered by VMware vSAN™

Abstract

This document provides best practices and architecture guidelines to design and build an SAP HANA environment on Hyperconverged Infrastructure (HCI) solutions based on Intel Cascade Lake based systems, VMware vSphere® and VMware vSAN™.  

The document provides information any hardware vendor can use as a starting point to build their own unique SAP HANA HCI solutions.

The vendor is responsible to define and select the best vSAN ready hardware components, VMware vSphere, and vSAN configuration for its unique solution. The HCI solution vendor is also responsible to certify its solution accordingly the SAP HANA Hardware Certification Hyperconverged Infrastructure certification scenario “HANA-HyCI 1.0 v1.2” in the SAP Integration and Certification Center (ICC).

This paper also includes information about the support processes and requirements of SAP HANA on vSAN.

Executive Summary

This section covers the business introduction and solution overview.

Overview

For more than 10 years, VMware has successfully virtualized SAP applications on x86 server hardware. It is a logical next step to virtualize and use software to define the storage for these applications and their data. SAP has provided production support for their applications on VMware vSphere for the last 10 years, and for HANA since 2014. 

SAP HANA based applications require dynamic, reliable, and high-performing server systems coupled with a predictable and reliable storage subsystem. The servers and storage must deliver the needed storage throughput and IOPS capacity at the right protection level and costs for all SAP HANA VMs in the environment. 

Storage solutions like large Fibre Channel SAN or standalone SAP HANA appliance configurations, are not able to quickly react to today’s changing and dynamic environment. Their static configurations have difficulty keeping pace with modern and resource-demanding applications. 

VMware vSAN, a software-defined storage solution, allows enterprises to build and operate highly flexible and application-driven storage configurations, which meets SAP HANA application requirements. Detailed SAP support information for VMware vSAN can be found in SAP Note 2718982 - SAP HANA on VMware vSphere and vSAN. A valid SAP s-user is required to view the note.

Audience

This paper is intended for SAP and VMware hardware partners who want to build and offer SAP HANA certified HCI solutions based on VMware technologies. 

Solution Overview

This section provides an overview of the technologies used in this solution.

SAP HANA on vSAN

SAP HANA is an in-memory data platform that converges database and application platforms, which eliminates data redundancy and improves performance with lower latency. SAP HANA architecture enables converged, online transaction processing (OLTP) and online analytical processing (OLAP) within a single in-memory, column-based data store that adheres to the Atomicity, Consistency, Isolation, Durability (ACID) compliance model. 

SAP environments with SAP HANA can use vSAN running on certified x86 servers to eliminate traditional IT silos of compute, storage, and networking with this hyperconverged solution. 

All intelligence and management are moved into a single software stack, allowing for a VM- and application-centric policy-based control and automation.

This brings better security, predictable performance, operational simplicity, and cost-effectiveness into SAP HANA environments. vSAN also simplifies the complexity of traditional IT silos for compute, storage, and networking, as it allows managing the SAP HANA storage just like compute resources.

HCI lowers costs by combining the economics and simplicity of local storage with the features and benefits of shared storage and can get used for SAP HANA Scale-Up and Scale-Out deployments.

Following figure summarizes the benefits of VMware based SPA HCI solutions.

 

Only SAP and VMware certified and tested components, like server and storage systems, are supported. These components must comply with the SAP HANA Hardware Certification Hyperconverged Infrastructure (HANA-HyCI) test scenario. 

Note: Only SAP and VMware HCI solution vendor offered solutions, that get SAP HANA HCI solution certified, are supported for SAP HANA workloads. SAP applications other than HANA are fully supported on a VMware vSAN cluster. No additional support notes or specific support processes are required from SAP when running other applications on vSAN, like an SAP application server.

Solution Components

An SAP HANA HCI solution based on VMware technologies is a fully virtualized and cloud ready infrastructure turnkey solution running on VMware vSphere and vSAN. All components like compute, storage, and networking are delivered together as an HCI vendor specific SAP HANA solution. It is based on a tightly integrated VMware software stack, where vSAN is part of the VMware ESXi™ kernel to efficiently provide great performing storage.

There are two ways to build a vSAN SAP HANA HCI cluster for an OEM:

  • Turnkey deployment using the existing vSAN/VCF appliances and modify these systems to be SAP HANA HCI ready.
  • Use the certified vSAN Ready Nodes and components and build an SAP HANA ready HCI solution.  

The solution consists of the following components:

  • VMware certified vSAN HW partner solutions, as listed on the VMware vSAN HCL (vSAN Appliances or Ready Nodes with vSAN Ready Components). 
  • SAP HANA supported server systems, as listed on the SAP HANA HCL.
  • VMware vSphere Enterprise Plus starting with version 6.5 update 2. For Cascade Lake based servers vSphere 6.7u3 or later is required. Review SAP note 2393917 for an updated list of supported vSphere versions.
  • VMware vSAN Enterprise starting with version 6.6. For Cascade Lake based servers vSAN 6.7u3 or later is required. Review SAP note 2718982 for an updated list of supported vSAN versions
  • VMware vCenter 6.5, 6.7u3, and 7.0u1. Note that vCenter corresponding versions are required.
  • HCI vendor implemented the vSAN HCI detection script provided by VMware (adds vSAN HCI identifier to the VM configuration file, see the SAP HANA HCI VMware vSAN Detection Script section for details).
  • HCI vendor SAP-integrated support process.

After the solutions are defined and built, SAP ICC validates and lists these solutions in Certified HCI Solutions on the SAP HANA Supported and Certified Hardware Directory section.

Figure 1 provides an overview of the solution components. 

Figure 1. SAP HANA HCI Solution Overview

SAP HANA HCI on vSAN—Support Status as of Note 2718982

SAP HANA HCI solutions are currently supported with the following VMware vSphere and vSAN versions and hardware:

  • VMware vSphere Hypervisor 6.5, 6.7, 6.7 U2/U3 and 7.0 U1
    • SAP HANA SPS 12 Rev. 122.19 (or later releases) for production single-VM and multi-VM use cases. See SAP Note 2393917 for detailed information and constraints. SAP HANA on vSAN has additional constraints, see next section.
    • vSphere 7.0 U1 is supported as of today for SAP HANA HCI solutions based on the vSphere 7.0 resource maximums, which are 6 TB RAM and up to 256 vCPUs per VM.
  • VMware vSAN 6.6, 6.7 (incl. U1, U2, and U3) and 7.0 U1[1]
    • vSAN 6.6, 6.7, 6.7 U1, 6.7 U2, 6.7 U3, 7.0 U1: Maximum 4 SAP HANA VMs on 2, 4, and 8 socket servers
    • Starting from above vSAN versions, the HCI vendor can select the appropriate vSAN version for their HCI solution.
    • The vSAN versions 6.6, 6.7 and 7.0 U1 are included in vSphere version 6.5 U2 and vSphere version 6.7, 6.7 U2/U3, and 7.0 U1 respectively.
    • Minimal 3, maximal 64 nodes per SAP HANA HCI vSAN cluster
    • 4 or more nodes are recommended for optimal failure resilience, even during maintenance and for cross data center wide, HA solutions are required.
    • No CPU sockets need to get reserved for vSAN. Note that vSAN will need, just like ESXi, system resources. Refer to VMware KB 2113954 for information about vSAN resource needs.
  • CPU architecture and VM maximums:
    • Any supported Intel Skylake or Cascade Lake processors with 2, 4, and 8 socket servers with existing VMware vSphere virtualization boundaries, for example:
      • max. 6 TB RAM and 256 vCPUs per SAP HANA Scale-Up VM, see SAP Note 2393917 for details. Also note that the actual usable vCPUs depend on the used CPU type: the current vSphere maximum vCPU number is 256.
    • The certified hardware for HCI solutions can be found on the Certified Hyper-Converged Infrastructure Solutions 
  • SAP HANA Deployment Options:
    • SAP HANA Scale-Up:
      • OLTP or mixed type workload up to 6 TB RAM and up to 256 vCPUs per VM
      • OLAP type workload: 
        • SAP sizing Class L: up to 3 TB RAM and up to 256 vCPUs per VM
        • SAP sizing Class M: up to 6 TB RAM and up to 256 vCPUs per VM
    • SAP HANA Scale-Out:
      • Up to 1 master and 7 worker nodes are possible, plus HA node(s). HA nodes should be added according to the cluster size.  FAQ: SAP HANA High Availability
      • SAP HANA Scale-Out system max. 16 TB memory with SAP HANA CPU L-Class* sizing:
        • Up to 2 TB of RAM and up to 256 vCPUs per Scale-Out node (VM), sizing up to SAP HANA CPU Sizing Class L type workloads
      • SAP HANA Scale-Out system max. 24 TB memory with SAP HANA M-Class* CPU sizing:
        • Up to 3 TB of RAM and up to 256 vCPUs per Scale-Out node (VM), sizing up to SAP HANA CPU Sizing Class M** type workloads
      • Additional requirements:
        • Scale-Out is only support by SAP with 4 or 8 socket servers and minimal 4-socket wide VMs that use up to 256 logical CPU threads, in the case of Cascade Lake 28 core CPUs this represents a maximal VM size with 224 vCPUs.
        • Only vSAN 6.7 U2/U3 is supported for SAP HANA Scale-Out
        • Only Cascade Lake based servers got SAP HANA Scale-Out validated and are therefore required.

Note: SAP Note 2718982 - SAP HANA on VMware vSphere and vSAN will be continuously updated with the latest support information without further notice. Above support status is from September 2020. Before planning an SAP HANA HCI vSAN based solution, check note 2718982 for any updates.  

The SAP HANA BW needed CPU Sizing Class can be found in the BW sizing report under the heading "RSDDSTAT ANALYSIS DETAILS". See SAP note 2610534 for details.

 

[1] vSAN 7.0 is not supported. 

Reference Architecture

This section introduces the scope, architecture, reference environment, hardware and software resources.

Overview

The purpose of the introduced vSAN for SAP HANA reference architectures is to provide a starting point for HCI solution vendors who want to build a vSAN based solution and to provide best practices and guidelines. 

The reference architecture is tested and validated with the SAP specified HCI tests and is known to be good with the shown SAP HANA configuration. 

Scope

This reference architecture:

  • Provides a technical overview of an SAP HANA on a VMware HCI solution
  • Shows the components used to test and validate vSAN with the SAP HCI tests and to meet the storage performance requirements of SAP HANA Tailored Datacenter Integration (TDI)
  • Explains configuration guidelines and best practices 
  • Provides information about vSAN/HCI solution sizing for SAP HANA VMs
  • Explains the deployment options from single DC to multiple suite configurations 

SAP HANA HCI on vSAN—Architecture Overview

vSAN Clusters consist of two or more physical hosts that contain either a combination of magnetic disks and flash devices (hybrid configuration) or all-flash devices (all-flash configuration). All vSAN nodes contribute cache and capacity to the vSAN distributed/clustered datastore and are interconnected through a dedicated 10 GbE (minimum) or better path redundant network. 

Note: In SAP HANA environments, four nodes are recommended for easier maintenance and Stretched Cluster readiness, but the minimum requirement of a three-host vSAN cluster for SAP HANA HCI can be used.

Figure 2 shows the high-level architecture of an SAP landscape running on a VMware-based SAP HANA HCI solution built upon several vSphere hosts and vSAN for a single data center configuration. Unlike with other SAP HANA HCI solutions, a VMware vSAN powered SAP HANA HCI solution can leverage all available ESXi host CPU and RAM resources. As shown in the figure, one host has enough compute resources available to support a complete host failover. 

Figure 2. SAP HANA Applications on vSAN High-Level Architecture

Storage policies for less demanding SAP HANA application servers and for IO demanding HANA VMs, as shown in Figure 2 (bronze, silver, and gold), will ensure that all VMs get the performance, resource commitments, failure tolerance, and quality of service as required policy-driven dynamic environment. With such an environment, changing the assigned storage policy in response to the changing needs of a virtualized application can directly influence the underlying infrastructure. The different storage policies allow changing the level of protection or possible IO limits without disturbing operation of a VM, allowing it to remain available and maintain the defined SLAs and performance KPIs.

SAP HANA HCI on vSAN—Scalability and Sizes

SAP HANA on vSphere is supported across a range of SAP HANA system size. The smallest size is a half socket CPU configuration with a minimum of 8 physical CPU cores and 128 GB of RAM. The largest size, as of today, is a 4 socket VM with 6 TB of RAM and 256* vCPUs.  Scale-Out SAP HANA deployments are also supported with up to eight 3 TB nodes. In total, up to 24 TB system memory can be deployed on SAP HANA vSAN HCI solutions for OLAP type workloads.

vSphere 6.7 U3 and 7.0 U1* as supported for vSAN provides the following maximums per physical host (table 1). For SAP HANA, only servers with up to 8 physical CPU sockets are currently supported.

*Actual usable number of vCPU depends on the selected CPU type. vSphere 7.0 U1 supports up to 6 TB and 256 vCPUs.

Table 1. Current vSphere HCI Supported Host Maximums

 

ESXi 6.7x (incl. U3) and 7.0 U1

Logical CPUs per host

768

Virtual machines per host

1,024

Virtual CPUs per host

4,096

Virtual CPUs per core

32

RAM per host

16 TB

NUMA Nodes per host

16 (SAP HANA only 8 CPU sockets)

Table 2 shows the maximum size of an SAP HANA VM and some relevant parameters such as virtual disk size and number of virtual NICs per VM.

Table 2.  vSphere Guest (VM) Maximums

 

ESXi 6.7x (incl. U3) and 7.0 U1

Virtual CPUs per virtual machine

256

RAM per virtual machine

6,128 GB

Virtual SCSI adapters per virtual machine

4

Virtual NVMe adapters per virtual machine

4

Virtual disk size

62 TB

Virtual NICs per virtual machine

10

Table 3 shows the vSAN maximum values: the component number or the size of a vSAN SAP HANA HCI cluster depends on the SAP HANA HCI vendor. The minimum SAP HANA supported cluster configuration is 3 vSAN cluster nodes, and up to 64 cluster nodes are supported by vSAN.

Table 3. vSAN Maximumx

 

VMware vSAN 6.7 (incl. U3) and 7.0 U1

vSAN ESXi host

vSAN Cluster

vSAN Virtual Machines

vSAN disk groups per host

5

 

 

Cache disks per disk group

1

 

 

Capacity tier maximum devices per disk group

7

 

 

Cache tier maximum devices per host

5

 

 

Capacity tier maximum devices

35

 

 

Components per vSAN host

9,000

 

 

 

 

 

 

Number of vSAN hosts in an All-Flash cluster

 

64

 

Number of vSAN hosts in a hybrid cluster

 

64

 

Number of vSAN datastores per cluster

 

1

 

 

 

 

 

Virtual machines per host

 

 

200

Virtual machine virtual disk size

 

 

62 TB

Disk stripes per object

 

 

12

Percentage of flash read cache reservation

 

 

100

Percentage of object space reservation

 

 

100

vSAN networks/physical network fabrics

 

 

2

Table 4 shows the size of SAP HANA VMs possible on a supported SAP HANA HCI solution for production use case with vSphere and vSAN 6.7 U3 and 7.0 U1. 

As of SAP note 2393917, the maximum size of a single SAP HANA VM is 6 TB of RAM and 224 vCPUs for Cascade Lake based server systems vCPUs. 256 vCPUs are the current vSphere maximum. SAP HANA VM can be configured as half-socket, full-socket, 2, 3, or 4-socket wide VMs. No CPU sockets need to be reserved for vSAN. If an SAP HANA system fits into these maximums, it can be virtualized with vSphere in a single VM. If it is larger, in terms of CPU or RAM needs, Scale-Out or SAP HANA data-tiering solutions can be used alternatively to meet the requirements. 

We have validated that 4 SAP HANA VMs can work consolidated on a vSAN configuration running on 2- or more-socket systems. For detailed information, see the Reference Test System Configuration section.

Table 4 shows the example CPU’s VM sizes that can be deployed on a vSAN SAP HANA HCI node.

Note: RAM must be reserved for ESXi. The RAM figures listed in Table 4 show the theoretical RAM maximums of a VM. The RAM for ESXi needs to get subtracted from this RAM size. No CPU sockets or other system resources need to get reserved for vSAN. VMware KB 2113954 provides details about the vSAN and memory consumption in ESXi.

Table 4. Example SAP HANA VM Configurations and Capacity of VMware Based HCI Solutions with the Selected Intel Cascade Lake CPUs

 

 

Intel Xeon Gold 6258R CPU

Intel Xeon Platinum 8280 CPU

Intel Xeon Platinum 8280L CPU

 Max supported RAM per CPU as of Intel datasheet1

1TB

1TB

4.5TB, SAP standard size is limited to 1.5 TB, with WBS more RAM possible!

CPU Cores per socket as of Intel datasheet1

28

28

28

Max. NUMA Nodes (CPU sockets) per server1

2

4, 8

4, 8

0.5 Socket SAP HANA VM (Half-Socket) 2

VM size:

14 pCores / 28 vCPUs, 128-480 GB RAM

VM size:

14 pCores / 28 vCPUs, 128-448 GB RAM

VM size:

14 pCores / 28 vCPUs, 128-704 GB RAM

Virtual SAPS capacity per VM3

28 vCPU VM = 34,000 vSAPS

28 vCPU VM = 36,000 vSAPS

28 vCPU VM = 36,000 vSAPS

1 Socket SAP HANA VM

VM size:

28 pCores / 56 vCPUs, 128-960 GB RAM

VM size:

28 pCores / 56 vCPUs, 128-896 GB RAM

VM size:

28 pCores / 56 vCPUs, 128-1408 GB RAM 

Virtual SAPS capacity per VM3

56 vCPU VM = 81.000 vSAPS

56 vCPU VM = 85,000 vSAPS

56 vCPU VM = 85,000 vSAPS

2 Socket SAP HANA VM

VM size:

56 pCores / 112 vCPUs, 128-1984 GB RAM

VM size:

56 pCores / 112 vCPUs, 128-1920 GB RAM

VM size:

56 pCores / 112 vCPUs, 128-2944 GB RAM

Virtual SAPS capacity per VM3

112 vCPU VM = 162,000 vSAPS

112 vCPU VM = 171,000 vSAPS

112 vCPU VM = 171,000 vSAPS

3 Socket SAP HANA VM

VM size:

84 pCores / 168 vCPUs, 128-2944 GB RAM

VM size:

84 pCores / 168 vCPUs, 128-4480 GB RAM 

Virtual SAPS capacity per VM3

-

168 vCPU VM = 256,000 vSAPS

168 vCPU VM = 256,000 vSAPS

4 Socket SAP HANA VM

VM size:

112 pCores / 224 vCPUs, 128-3968 GB RAM

VM size:

112 pCores / 224 vCPUs, 128-6016 GB RAM 

Virtual SAPS capacity per VM3

-

224 vCPU VM = 342,000 vSAPS

224 vCPU VM = 342,000 vSAPS

Review the Intel published CPU datasheets for all CPU data. Depending on the selected CPUs the supported memory per CPU socket and number of CPU cores may differ from the shown example configurations. Also review the SAP supported memory configuration for a specific CPU and workload. The shown figures here are up to the maximum Intel specific specified values and do not consider specific SAP workload limitations.

2The half-socket RAM figures listed in the table show even configured half-socket VMs. The SAPS capacity gets reduced by 15% because of NUMA node sharing when compared to a VM running alone on a NUMA node. The maximum RAM of a half-socket configuration can be as large as maximum RAM installed per NUMA node divided by 2. For full socket VMs, the memory can be up to the installed per CPU socket multiplied by the number of used CPU sockets. The shown memory considers a 64 GB RAM virtualization overhead for 2-socket based systems and 128 GB RAM for 4-socked bases systems. This overhead is the memory vSphere and vSAN are utilizing. The pre-occupied memory for vSAN and vSphere may be higher and the exact amount of consumed memory can only get determined once configuring a VM on the actual configuration. VMware KB 2113954 provides details about the vSAN and memory consumption in ESXi.

3The shown vSAPS figures are based on SAP SD benchmark cert. 2020015 (Intel 6258R) and cert. 2019023 (Intel 8280). The shown numbers got reduced by a 10% virtualization cost buffer. The VMs use hyperthreading; therefore, make sure to configure the VMs accordingly. 

 

In addition to Scale-Up SAP HANA system RAM sizes of up to 6 TB, Scale-Out deployments are supported with up to 3 TB VMs, distributed over up to 8 SAP HANA HCI plus HA nodes as required. This provides for an OLAP type SAP HANA instance up to 24 TB in total system RAM by utilizing the Scale-Out deployment model.

Figure 3 shows a 4+1 SAP HANA HCI BW cluster. As of today, up to eight[1] 3 TB Scale-Out nodes (VMs) are supported with vSAN. The required SAP HANA “Shared Directory” gets mounted directly to SAP HANA VMs via NFS from an SAP HANA HCI external NFS server.

Figure 3. SAP HANA Scale-Out Example Cluster on vSAN (HCI servers)


[1] This is a support rather than technical limitation.

VMware HA to Protect Scale-Up and Scale-Out SAP HANA Systems

VMware vSphere HA delivers a vSphere built-in high availability solution, which can be used to protect SAP HANA. VMware HA provides unified and cost-effective failover protection against hardware and operating system outages and works with a vSAN data store just like it does with any other supported shared storage solution with VMFS. 

VMware HA protects SAP HANA Scale-Up and Scale-Out deployments without any dependencies on external components, such as DNS servers, or solutions, such as the SAP HANA Storage Connector API or STONITH scripts.

Note: See the Architecture Guidelines and Best Practices for Deployments of SAP HANA on VMware vSphere for detailed information about how VMware HA is used to protect any SAP HANA deployment on vSphere and how to install an SAP HANA Scale-Out system on VMware vSphere.

In the event of a hardware or OS failure, the entire SAP HANA VM is restarted either on the same or other hosts in the vSphere cluster. Because all virtual disks, such as OS, data, and log VMDKs are stored on the shared vSAN datastore, no specific storage tasks, STONITH scripts, or extra cluster solutions are required.

Figure 4 shows the concept of a typical n+1 configuration, which is the standard deployment of an SAP HANA HCI solution. It is recommended deploying an n+2 configuration for easy maintenance and to protect your systems against a failure while you perform maintenance.

Figure 4. VMware HA Protected SAP HANA VMs in an n+1 vSAN Example Cluster Configuration

Figure 5 shows the same configuration with an active-active cluster. This ensures that all hosts of a vSAN cluster are used by still providing enough failover capacity for all running VMs in the case of a host failure.

Figure 5. VMware HA Active-Active n+1 vSAN Example Cluster Configuration

The same concept is also used for SAP HANA Scale-Out configurations. Note that for SAP HANA Scale-Out, only 4-socket or 8-socket servers are supported. The minimum SAP HANA VM must be configured with 4 virtual CPU sockets respectively. Figure 6 shows a configuration of 3 SAP HANA VMs (1 x master with 2 worker nodes) running on the virtual SAP HANA HCI platform. The NFS mounted SAP HANA shared directory is not part of the vSAN cluster and needs to be provided as an vSAN external NFS service*.

 

* vSAN 7.0 U1 File Service (NFS) is not supported.

Figure 6. SAP HANA Scale-out VMware HA n+1 vSAN Example Cluster Configuration

Unlike physical deployments, there are no dependencies on external components, such as DNS servers, SAP HANA Storage Connector API, or STONIT scripts. VMware HA will simply restart the failed SAP HANA VM on the VMware HA/standby server. HANA shared is mounted via NFS, just as recommended with physical systems and will move with the VM that has failed. The access to HANA shared is therefore guaranteed.

Note: VMware HA can protect only against OS or VM crashes or HW failures. It cannot protect against logical failures or OS file system corruptions that are not handled by the OS file system. vSAN 7.0 U1 File Service (NFS) are yet not supported.

Reference Test System Configuration

This section introduces the resources and configurations including:

  • Reference environment 
  • Hardware resources
  • Software resources

Note: The shown reference configuration should be treated as an example but not as a requirement. The HCI solution vendor may or may not follow the shown example configurations and select different components. The only requirements are that the SAP HANA HCI KPIs are met and that only supported/certified components are used.

Reference Environment

To ensure that a vSAN based HCI solution is SAP HANA ready, SAP provided test tools and test cases (HANA-HyCI 1.0 v1.2 test scenario). These test tools validate performance, scalability, and consistency. Under the “Certified Hyper-Converged Infrastructure Solutions“, a complete set of vSAN supported configurations from different vendors are listed. For example, this link provides a list for Cascade Lake and vSAN supported configurations.

The reference configurations listed on the next pages are validated by VMware and provide an example for Scale-Up and Scale-Out configurations. The Scale-Up example configurations are based on 2- and 4-socket Cascade Lake Intel processors, configured as a 3-host and 4-host vSAN clusters with 2 disk groups and 4 disk groups per server. The configurations had NVMe and Intel Optane drives as caching layer, and SSD based capacity disks. With both configurations, it is possible to meet the SAP HANA KPIs defined for each configuration. The Scale-Out test configuration ran on a 4-host vSAN cluster. Each host had 3 TB memory, 4 CPU sockets, and 4 disk groups installed and configured.

Hardware Resources

Two vSAN clusters were configured in the 4-host and 3-host environment respectively:

  • One vSAN SAP HANA HCI environment with 4 hosts, having a combination of three 4 socket Cascade Lake and one 2 socket Cascade Lake processors configured with 4 disk groups per each vSAN host. 
  • One vSAN SAP HANA HCI environment with four 2-socket based Cascade Lake processors with 2 disk groups per each vSAN host. 

2-Socket/2-Disk Group SAP HANA HCI vSAN Based Example

The 2-socket/2-disk group test environment had four vSphere ESXi servers configured and used direct-attached SAS and NVMe devices inside the hosts to provide a vSAN datastore that meets the SAP HANA HCI test requirements and KPIs for two SAP HANA VMs per server.

The fourth node was part of the vSAN datastore and provided HA capabilities for the total 6 running SAP HANA VMs in the vSAN cluster.

Each vSphere ESXi server had two disk groups each consisting of one 1.46 TB NVMe device for caching and four capacity tier 3.49 TB SAS SSDs. The raw capacity of the vSAN datastore of this 4-node vSAN cluster was for data around 112 TB. The vSAN cluster limit is up to 64 cluster nodes. The vSAN cluster node number that an HCI vendor supports is up to the vendor.

Two 25 Gbit network adapters were used for vSAN and another network traffic in active-standby failover manner. 

Each ESXi Server in the vSAN Cluster had the following configurations as shown in the following table, based on this configuration, up to two running SAP HANA VMs per HCI server.

PROPERTY

SPECIFICATION

ESXi server model

HW Vendor SAP HANA Supported Cascade Lake Server Model

ESXi host CPU

2 x Intel(R) Xeon(R) Gold 6248 CPU, 2 Sockets, 20 Physical Cores/Socket.

ESXi host RAM

1.5 TB 

Network adapters

2 x 25 Gbit as a shared network for vSAN, SAP application and admin network traffic.

See the network section of this guide for the recommended network configurations. SAP HANA HCI Network Configuration

Storage adapter

4 x 8 channel PCI-EXPRESS 3.0 SAS 12GBS/SATA 6GBS SCSI controller

Storage configuration

Disk Groups: 2

Cache Tier: 2 x 750 GB Intel Optane DC 4800X Series

Capacity Tier: 12 X 3.8 TB 3840 Micron 5200 TCG-E

Linux LVM configuration

Used Linux LVM for log and data.

Used 4 VMDK disks in LVM each (total 8) to create the needed log and a data volume for SAP HANA.

4-Socket/4-Disk Group SAP HANA HCI vSAN Cluster - Example 1

The 4-socket test environment had three vSphere ESXi servers configured and used direct-attached SSDs and NVMe inside the hosts to provide a vSAN datastore that meets the SAP HANA HCI test requirements and KPIs. 

The third node was part of the vSAN datastore and provided HA capabilities for the total 8 running SAP HANA VMs. Adding the fourth node would allow you to run more SAP HANA VMs on the vSAN datastore. The vSAN cluster limit is up to 64 cluster nodes. The vSAN cluster node number that an HCI vendor supports is determined by the vendor.

Each vSphere ESXi server had four disk groups with each consisting of 800 GB NVMe devices for caching and four capacity tier 890GB SAS SSDs. The raw capacity of the vSAN datastore of this 3-node vSAN cluster was for data around 43 TB. The vSAN cluster limit is up to 64 cluster nodes.

Two 25 Gbit network adapters were used for vSAN and other network traffic in active-standby failover manner. 

Review the network section for details about how the network cards impact the number of SAP HANA VMs per host and the sizing section for how many SAP HANA VMs can be deployed on a Skylake based server system.

Each ESXi Server in the vSAN Cluster had the following configuration as shown in Table 5. This configuration supports up to four SAP HANA VMs per HCI server.

Table 5.  ESXi Server Configuration of the 4-socket Server SAP HANA on 3-node HCI vSAN Cluster

PROPERTY

SPECIFICATION

ESXi server model

HW Vendor SAP HANA Supported Cascade Lake supported Model

ESXi host CPU

4 x Intel(R) Xeon(R) Platinum 8280M CPU 2.70GHz , 4 Sockets, 28 Physical Cores/Socket

ESXi host RAM

1.5 TB 

Network adapters

2 x 25 Gbit as a shared network for vSAN, SAP application and admin network traffic.

See the network section of this guide for recommended network configurations.

Storage adapter

SAS 12GBS/SATA 6GBS SCSI controller

Storage configuration

Disk Groups: 4

Cache Tier: 4 x Express Flash PM1725a 800GB SFF

Capacity Tier: 16 x  SAS  894 GB SSD 

4-Socket/4-Disk Group SAP HANA HCI vSAN Cluster Example 2

The 4-socket test environment had three vSphere ESXi servers configured and used direct-attached SSDs and NVMe inside the hosts to provide a vSAN datastore that meets the SAP HANA HCI test requirements and KPIs. 

Each ESXi Server in the vSAN Cluster had the following configuration as shown in Table 6. This configuration supported running up to four SAP HANA VMs per HCI server.

Table 6.  ESXi Server Configuration of the 4-socket Server SAP HANA on 3-node HCI vSAN Cluster

 

PROPERTY

SPECIFICATION

ESXi server model

HW Vendor SAP HANA Supported Cascade Lake supported Model

ESXi host CPU

4 x Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz, 4 Sockets, 28 Physical Cores/Socket

ESXi host RAM

1.5 TB 

Network adapters

2 x 100 Gbit as a shared network for vSAN, vMotion, SAP application and admin network traffic. Each network is carved out on a separate VLAN.

See the network section for recommended network configurations.

Storage adapter

SAS 12GBS/SATA 6GBS SCSI controller

Storage configuration

Disk Groups: 4

Cache Tier: 4 x NVMe 750G 

Capacity Tier: 16 x 1.75TB SSD 

Linux LVM configuration

Linux LVM used with 8 VMDK for Data partition and  8 VMDK for Log partition

Socket/4-Disk Group SAP HANA Scale-Out on HCI vSAN Cluster Example Configuration

The 4-socket test environment had four vSphere ESXi host servers in the vSAN cluster. Each host had 4 Intel Optane NVMe 750G for caching layer, 3TB memory and class M based CPU. The SSD based capacity drives provided the vSAN datastore for SAP HANA scale out configuration.  Up to 8 hosts (7+1 HA) can be configured for the scale out environment.

The shared partition for NFS can be provided by vSAN data store with a VM running on the HA node or a separate NFS based storage. In this config a separate NFS based storage was used for shared partition. BWH benchmark was run and a separate ESXi host running NetWeaver VM is used as a driver to do user throughput testing. 

Each vSphere ESXi server had four disk groups each consisting of 1.6TB devices for caching and four capacity tier 4TB SAS SSDs. 

Two 25 Gbit network adapters were used for vSAN and other network traffic in active-standby failover manner. 

Review the network section for details on how the network cards impact the number of SAP HANA VMs per host and the sizing section for how many SAP HANA VMs can be deployed on a Skylake based server system.

Each ESXi Server in the vSAN Cluster had the following configuration as shown in Table 8. This configuration supported running up to four SAP HANA VMs per HCI server.

 

Table 7. SAP HANA Scale-Out Example Configuration

PROPERTY

SPECIFICATION

ESXi server model

HW Vendor SAP HANA Supported Cascade Lake  Server Model

ESXi host CPU

4 x Intel(R) Xeon(R) Gold 20C CPU 2.10GHz , 4 Sockets, 20 Physical Cores/Socket

ESXi host RAM

3 TB 

Network adapters

2 x 25 Gbit as a shared network for vSAN, SAP application, and admin network traffic.

See the network section of this guide for the recommended network configurations.

Storage adapter

Dell HBA330 Adapter SAS 12GBS/SATA 6GBS SCSI controller

Storage configuration

Disk Groups: 4

Cache Tier: 4 NVMe p4610 1.6TB 

Capacity Tier: 16 x SAS p4510 4TB SFF

Linux LVM configuration

Linux LVM used with 8 VMDK for Data partition and 8 VMDK for Log partition

Software Resources

Table 8 shows the VMware related software resources used in this solution. From the SAP side, both SAP HANA version 1.0 and 2.0 are supported.

Table 8.  Software Resources

SOFTWARE

VERSION

PURPOSE

VMware vCenter Server

6.7 U3 and 7.0 U1

VMware vCenter Server provides a centralized platform for managing VMware vSphere environments.

vSphere ESXi

6.7 U3 and 7.0 U1

vSphere ESXi cluster to host virtual machines and provide vSAN Cluster. 

VMware vSAN

6.7 U3 and 7.0 U1

Software-defined storage solution for hyperconverged infrastructure

SAP HANA HCI on vSAN Configuration Guidelines

This chapter introduces the configuration guidelines.

Overview

Selecting the correct components and a proper configuration is vital to achieving the performance and reliability requirements for SAP HANA. The server and vSAN configurations (RAM and CPU) determine how many SAP HANA VMs and which VMs are supported. 

The vSAN configuration specifics such as the number of disk groups, selected flash devices for the caching and capacity storage tier or the vSAN network configuration define how many SAP HANA VMs a vSAN based datastore can support. 

SAP HANA HCI Network Configuration

To build an SAP HANA ready HCI system, dedicated vSAN networks for SAP application, user traffic, admin and management are required. Follow the SAP HANA Network Requirements document to decide how many additional network cards have to be added to support a specific SAP HANA workload on the servers.

Table 9 provides an example configuration for an SAP HANA HCI configuration based on dual port network cards.

Table 9.  Recommended SAP HCI on vSphere Network Configuration

 

 

ADMIN AND VMWARE HA NETWORK

APPLICATION SERVER NETWORK

vMotion NETWORK

vSAN NETWORK

BACKUP NETWORK

SCALE-OUT INTERNODE NETWORK

SYSTEM REPLICATION NETWORK

Host Network Configuration 
(all networks are dual port NICs, 
configured as failover teams)

Network Label

Admin

App-Server

vMotion

vSAN

Backup

Scale-Out

HANA-Repl

Recommended MTU Size

Default (1500)

Default (1500)

9000

9000

9000

Bandwidth

1 or 10 GbE

1 or 10 GbE

>= 10 GbE*

 

>= 10 GbE*

Physical NIC

0

1

 

 

OPTIONAL

Physical NIC port

1

2

1

2

VLAN ID#

100

101

200

201

202

203

204

VM Guest Network Cards

Virtual NIC (inside VM)

0

1

-

-

2

2

3

*The selected network card bandwidth for vSAN directly influences the number of SAP HANA VMs that can be deployed on a host of the SAP HANA HCI vSAN cluster. For instance, a network exceeding 10 GE is needed if more than two SAP VMs should be deployed per host.

In addition to using a high bandwidth and low latency network for vSAN, it is also strongly recommended using a VMware vSphere Distributed Switch™ for all VMware related network traffic (such as vSAN and vMotion).  

A Distributed Switch acts as a single virtual switch across all associated hosts in the data cluster. This setup allows virtual machines to maintain a consistent network configuration as they migrate across multiple hosts.

The vSphere Distributed Switch in the SAP HANA HCI solution uses at least two 10 GbE dual port network adapters for the teaming and failover purposes. A port group defines properties regarding security, traffic shaping, and NIC teaming. It is recommended to use the default port group setting except the uplink failover order that should be changed as shown in Table 10. It also shows the distributed switch port groups created for different functions and the respective active and standby uplink to balance traffic across the available uplinks. 

Table 10 and Table 11 show an example of how to group the network port failover teams. It does not show the optional networks needed for VMware or SAP HANA system replication or Scale-Out internode networks. Additional network adapters are needed for these networks.

Table 10. Minimum ESXi Server Uplink Failover Network Configuration (if all NICs have the same bandwidth to allow physical NIC failure recovery)

PROPERTY

VLAN*

ACTIVE UPLINK

STANDBY UPLINK

Admin plus VMware HA Network

100

Nic0-Uplink1

Nic1-Uplink1

SAP Application Server Network

101

Nic0-Uplink2

Nic1-Uplink2

vMotion Network

200

Nic1-Uplink1

Nic0-Uplink1

vSAN Network

201

Nic1-Uplink2

Nic0-Uplink2

*VLAN id example, final VLAN numbers are determined by the network administrator.

Table 11. Minimum ESXi Server Uplink Failover Network Configuration (if NICs have different bandwidth, with no physical NIC failover possiblility)

PROPERTY

VLAN*

ACTIVE UPLINK

STANDBY UPLINK

Admin plus VMware HA Network

100

Nic0-Uplink1

Nic0-Uplink2

SAP Application Server Network

101

Nic0-Uplink2

Nic0-Uplink1

vMotion Network

200

Nic1-Uplink1

Nic1-Uplink2

vSAN Network

201

Nic1-Uplink2

Nic1-Uplink1

We used different VLANs to separate the VMware operational traffic (for example: vMotion and vSAN) from the SAP and user-specific network traffic.

VMware ESXi Server: Storage Controller Mode

Server storage or RAID controller normally support both pass-through and RAID mode. 

In vSAN environments, only use controllers that support pass-through mode, as it is a requirement to deactivate RAID mode and use only pass-through mode. 

vSAN Disk Group Configuration

Depending on the hardware configuration, like the number of CPUs and flash devices, minimal two or more disk groups should be configured per host. More disk groups improve the storage performance. The current disk group maximum is five disk groups per vSAN cluster node.

 

The cache disk type should be a low-latency high-bandwidth SAS SSD or an NVMe device. The capacity should be 600 GB or larger. Review VMware vSAN design considerations for flash caching devices and the All-Flash Cache Ratio Update document for details.

In our 2 and 4-socket server reference configuration, we used two and four disk group configurations to optimally use the server hardware components installed. The number of disk groups should be selected depending on how many SAP HANA VMs will be deployed on a single vSAN cluster node. In some test configurations, we noticed that a 2-disk group configuration with NVMe for caching and four capacity disks was enough to meet the SAP HANA KPIs for two SAP HANA VMs per server. If four SAP HANA VMs should be deployed on a server, we recommend using a 4-disk group configuration. In addition, distribute the NVMe across all the CPUs wherever possible, if multiple NVMe are located on single CPU link, ensure that separate ports are used. As noted, the HW vendors are free to decide which configuration they want to certify.

In our Cascade Lake 2-socket/2-disk group and 4-host server vSAN cluster configuration, we used a write optimized 2x 750 GB Intel Optane DC 4800X Series for caching and 6 X 3.8TB 3840 Micron 5200 TCG-E SAS SSDs for the capacity tier per disk group. In total, we used 12 SSDs per server, all connected on one vSAN certified SCSI adapter. With this vSAN configuration, we were able to validate two SAP HANA VMs per server successfully. 

In the next 4-disk group configuration, we used 4-socket in a three node vSAN cluster configuration. We used an 800GB NVMe device for caching and four SAS 900GB SSDs for the capacity tier per disk group. With this vSAN configuration, we were able to validate 4 SAP HANA VMs per server. Additional tests with other configurations also found that fewer capacity SSDs are needed to pass the tests by using NVMes in the caching tier. The number of capacity SSDs depends mainly on the size of the SAP HANA VMs that will run on the HCI solution.

Figure 7 shows a single server of our 4-disk group test configuration in a graphical way.

Figure 7. 4-disk Group Configuration Example of Our SAP HANA HCI Test Configuration

Replacing the SAS SSDs for caching with a higher bandwidth, low latency NVMe devices helped us to reduce the number of vSAN disk groups from four to two, while still maintaining the SAP HANA storage KPIs for two SAP HANA VMs per server. 

Note: If 4 or more SAP HANA VMs are supported per vSAN node, it is recommended to use at least a 4-disk group configuration.

Figure 8. 2-disk Group Configuration of Our SAP HANA HCI Test Configuration

Note: The number of capacity drives per disk group and the number of disk groups per host impact the maximum number of SAP HANA VMs per server. 

vSAN Storage Configuration

All vSAN disk groups are configured as an into single cluster-wide shared datastore. The actual storage configuration for a VMDK file will be defined by a specific vSAN storage policy that fits its usage (OS, log, or data). For SAP HANA, we use the same storage policy for log and data. For the VMDK that contains the OS or SAP HANA binaries, the storage policy can be modified to use erasure coding to optimize capacity instead of performance.

vSAN Storage Policy

The SAP HANA HCI solution is a very specific configuration and is optimized for the high-performance demands of an SAP HANA database. Just like an SAP HANA appliance, the storage subsystem is optimized to meet the SAP HANA KPIs for reliability, consistency, and performance. These KPIs are validated, tested, and finally certified by SAP. 

Depending on the actual vSAN hardware configuration, a vSAN storage policy is selected and applied to the SAP HANA OS, log and data disks. The vSAN policy is one of the parameters that determine the number of HANA VMs that can be supported on a server.

Table 12 and Table 13 show the vSAN storage policy we used for our testing. This storage policy should be applied on both the data disks and the log disk of each SAP HANA Database VM.

For non-performance critical volumes, the vSAN default storage policy or a custom RAID 5 storage policy can be used to lower the storage capacity need of these volumes. 

For example, a VM protected by a primary level of failures to tolerate value of 1 with RAID 1 requires twice the virtual disk size, but with RAID 5, it requires 1.33 times the virtual disk size. For RAID 5, you need a minimum of 4 vSAN nodes per vSAN cluster.

The number of stripes configured should equal the number of capacity devices contained in a host. Note that the maximum number of stripes that can be configured in a storage policy is 12. 

Table 12. vSAN Storage Policy Settings for Data and Log for a Configuration with 12 or More Capacity Disks 

STORAGE CAPABILITY

SETTING

Failures tolerance method (FTM)

RAID 1 (Mirroring)

Number of failures to tolerate (FTT)

1

Minimum hosts required

3

Number of disk stripes per object

12

Flash read cache reservation

0%

Object space reservation

100%

Disable object checksum

No

Table 13 shows the vSAN policy for a 2-disk group vSAN cluster configuration.

Table 13. vSAN Storage Policy Settings for Data and Log for a Configuration with 8 Capacity Disks

STORAGE CAPABILITY

SETTING

Failure Tolerance Method (FTM)

RAID 1 (Mirroring)

Number of FTT

1

Minimum hosts required

3

Number of disk stripes per object

8

Flash read cache reservation

0%

Object space reservation

100%

Disable object checksum

No

Table 14 shows the vSAN policy for less performance critical disks like OS or SAP HANA shared.

Note: You need a minimum of 4 vSAN nodes to use RAID 5, and minimum of 6 vSAN nodes for RAID 6.

Table 14. vSAN Storage Setting for OS or Other Less Performance Critical Volumes

STORAGE CAPABILITY

SETTING

Failure tolerance method (FTM)

RAID 5/6 (Erasure Coding)

Number of FTT

1

Minimum hosts required

4

Number of disk stripes per object

1

Flash read cache reservation

0%

Object space reservation

0%

Disable object checksum

No

SAP HANA HCI Storage Planning and Configuration

When planning a virtualized SAP HANA environment running on vSAN, it is important to understand the storage requirements of such an environment and the applications running on it. It is necessary to define the storage capacity and IOPS needs of the planned workload. 

Sizing a storage system for SAP HANA is very different from storage sizing for SAP classic applications. Unlike with SAP classic applications, where the storage subsystem is sized regarding the IOPS requirements of a specific SAP workload, the SAP HANA storage is defined by the specified SAP HANA TDI storage KPIs. These data throughput and latency KPIs are fixed values per SAP HANA instance. 

Depending on how many SAP HANA VMs a vendor is supporting on an SAP HANA HCI cluster, the vSAN datastore must be able to maintain the performance KPIs and must be able to provide the storage capacity all installed SAP HANA VM on the vSAN cluster.

VMware has successfully tested a 4-socket 3-node vSAN based SAP HANA HCI configuration with an 8 SAP HANA VM configuration (4 SAP HANA VMs per one active node) and a 2-socket 4-node cluster with 2 SAP HANA VM per server configuration. These configurations represent one SAP HANA VM per socket, leaving the third or fourth node in the cluster used as an HA node. This leaves enough capacity available to support a full node failure at any time.

SAP HANA hardware configuration check tool (HWCCT) is the tool available only for SAP partners and customers that test a system and measure it against the currently valid storage KPIs. The tool and the documentation can be downloaded from SAP with a valid SAP user account. See SAP note 1943937

SAP HANA Storage Capacity Calculation and Configuration

All SAP HANA instances have a database log, data, root, local SAP, and shared SAP volume. The storage capacity sizing calculation of these volumes is based on the overall amount of memory needed by SAP HANA’s in-memory database. 

SAP has defined very strict performance KPIs that have to be met when configuring a storage subsystem. These storage performance KPIs are the leading sizing factor. How many capacity devices are used to provide the required I/O performance and latency depends on the HCI vendor hardware components. 

The capacity of the overall vSAN datastore must be large enough to store the SAP HANA data, log, and binaries by providing the minimal FTT=1 storage policy. SAP has published several SAP HANA specific architecture and sizing guidelines, like the SAP HANA Storage Requirements. The essence of this guide is summarized in Figure 9 and Table 15, which can be used as a good start for planning the storage capacity needs of an SAP HANA system. It shows the typical disk layout of an SAP HANA system and the volumes needed. The volumes shown in the document should correspond with VMDK files to ensure the flexibility for the changing storage needs (the increased storage capacity most likely).

Figure 9. Storage Layout of an SAP HANA System. Figure © SAP SE, Modified by VMware

Table 15 determines the size of the different SAP HANA volumes. It is based on the SAP HANA Storage Requirements document. Some of the volumes like the OS and usr/sap volumes can be connected and served by one PVSCSI controller, others like the LOG and data volume are served by dedicated PVSCSI controllers to ensure high IO bandwidth and low latency.

Table 15.  Storage Layout of an SAP HANA System

 

Volume

VMDK

SCSI Controller

VMDK Name

SCSI ID

Sizes as of SAP HANA Storage Requirements

/(root)

ü

PVSCSI Contr. 1

vmdk01-OS-SIDx

SCSI 0:0

Minimum 40 GiB for OS 

usr/sap

ü

PVSCSI Contr. 1

vmdk01-SAP-SIDx

SCSI 0:1

Minimum 50 GiB for SAP binaries 

hana/shared

ü

PVSCSI Contr. 1

Vmdk02-SHA-SIDx

SCSI 0:2

Minimum 1x RAM max. 1 TB

hana/data

ü

PVSCSI Contr. 2

Vmdk03-DAT1-SIDx

Vmdk03-DAT2-SIDx vmdk03-DAT3-SIDx

SCSI 1:0

SCSI 1:1

Minimum 1 x RAM

Note: If you use multiple VMDKs, use for example Linux LVM to build one large data disk.

hana/log

ü

PVSCSI Contr. 3

Vmdk04-LOG1-SIDx

Vmdk04-LOG2-SIDx

SCSI 2:0

SCSI 2:1

[systems <= 512GB] log volume (min) = 0.5 x RAM 

[systems >= 512GB] log volume (min) = 512GB

hana/backup

(optional)

ü

PVSCSI Contr. 4

Vmdk05-BAK-SIDx

SCSI 3:1

Size backups >= Size of HANA data+ size of redo log.

The default path for backup is (/hana/shared). Change the default value when a dedicated backup volume is used.

Figure 10 shows the default SAP HANA storage layout as documented in SAP’s HANA storage white paper, in a graphical view with the dedicated VMDKs per OS, HANA shared, log, and data volumes. 

Figure 10. Dedicated VMDK Files per SAP HANA Volume

A “pure” VMDK solution provides the easiest way to get started and is enough to achieve the SAP HANA HCI performance and scalability KPIs.

Usage of Linux Logical Volume Manager (LVM)

Although we did not use Linux LVM with our 4-socket, 3 node cluster tests, the usage of LVM is supported and can be used if desired. Using Linux LVM can help to leverage more vSAN resources and balance the VMDK components distribution on vSAN. On the other hand, using LVM is adding configuration complexity. It is determined by the HCI solution vendor to use LVM or not.

Figure 11 shows an example how LVM can be used for the SAP HANA LOG and DATA volumes. We used this configuration with our 4-disk group SAS cache and 2-disk group configurations. Linux LVM used with 8 VMDK for data partition and a min 4 to 8 VMDK for log partition.

Note that the number of VMDK disks used for LVM should not exceed the overall number that physical capacity disks used in a host. For example, if 6 capacity disks are used in a server, 8 VMDK disks for LVM would require because two disks will have to store the objects that represent two VMDK files. This would result in an unbalanced configuration. Whereas 8 VMDK disks would be used if 8 or more capacity disks are installed. For example, if 6 capacity disks are used in a server, up to 6 VMDK disks for LVM would be ideal. 

Figure 11. Usage of LVM Managed Volumes

vSAN Storage Capacity Calculation

To determine the overall storage capacity per SAP HANA VM, summarize the sizes of all specific and unique SAP HANA volumes as outlined in Figure 9 and Table 15. 

To determine the overall vSAN datastore capacity required, without cache tier, it is required to multiply the above SAP HANA volume requirements with the amount of VMs that can run on the SAP HANA HCI cluster.

Simplified calculation: 

vSAN datastore capacity = total HANA VM x (OS + USR/SAP + SHARED + DATA + LOG) x 2 (FTT=1, RAID 1) + 30% Slack space. 

Example: 

A cluster with 3 servers with 18 TB RAM and 8 SAP HANA VMs in total, one server is HA -> 12 TB of SAP HANA data needs to be stored.

8 x (10 GB + 60 GB + 1 TB + 12 TB / 8 VMs + 512 GB) x 2 x 1.3 = 8 x 3,142 GB x 2 x 1.3 = ~65.4 TB

We have over 138 TB of vSAN raw datastore capacity available with our example 3-node 4-socket SAP HANA HCI configuration, which is much more than the amount required for up to 8 SAP HANA VMs with a total up to 12 TB of SAP HANA system RAM capacity. The additional space can be used to store optional backups. This lowers the TCO since no additional nearline backup storage is needed. Nevertheless, moving the backup files to a permanent external backup location is recommended and required.

SAP HANA HCI VMware vSAN Detection Script

SAP requested to implement a method to detect if an SAP HANA VM runs on an SAP HANA HCI solution. VMware and SAP worked together to develop a script that allows SAP to detect at the runtime if an SAP HANA VM is running on a vSAN datastore. 

The VMware vSAN detection script is available on request from VMware. Contact the author or contributors of this document for the script. The script needs to be installed by the SAP HANA HCI solution vendor on their VMware based SAP HANA HCI solution systems.

The script is a Python script that will be executed every 5 minutes on the vSAN hosts that build the SAP HANA HCI solution and will add the following information to the VM configuration file:

  • "guestinfo.vsan.enabled": True/False,
  • "guestinfo.SDS.solution": esxi_version, which is also the vSAN release version
  • "guestinfo.vm_on_vsan": True/False

This data will be parsed by the SAP CIM provider and SAP support tools like SAPsysinfo, and it detects if an SAP HANA HCI on VMware vSAN solution is used for this specific VM. Details about the VMware SAP key monitoring metrics can be found in OSS Note 2266266—Key Monitoring Metrics for SAP on VMware vSphere (version 5.5 u3 and higher).

Prerequisites:

  • Enable SSH for the time of installation. Deactivate it after the installation of the script.
  • To simplify the installation process, it is recommended to have identical password of the super user on all the ESXi servers.

Installation and execution of the health check installation script

  1. Enable SSH on the vSphere ESXi hosts systems.
  2. Place the installation script and setInfo.vib into the SAME directory of any Linux/MacOS system.
  3. Run ./install_vib_script_remotely.sh and type in the following information as prompted:

root@Centos [ ~/hana_health_check ]# ./install_vib_script_remotely.sh -h

Please specify the one of the following parameters:

-i or --install: install the vib on specified hosts

-r or --remove: remove the vib on specified hosts

-u or --update: update the vib on specified hosts

-h or --help: display this message

  1. To install the vib on the specified:

[w3-dbc302:hana_vib]$./install_vib_script_remotely.sh -i

Please specify the hosts(Separate by whitespace):

10.xx.xx.xx 

login as: root

Password:

 

Validating connectivity to 10.xx.xx.xx

10.xx.xx.xx connectivity validated

 

Uploading setInfo_v1.3.vib to 10.xx.xx.xx

Successfully uploaded setInfo_v1.3.vib to 10.xx.xx.xx:/tmp

 

Installing VIB in 10.xx.xx.xx

Host acceptance level changed to 'CommunitySupported'.

Installation Result:

   Message: Operation finished successfully.

   Reboot Required: false

   VIBs Installed: VMware_bootbank_setInfo_1.3.0-6.7.0

   VIBs Removed:

   VIBs Skipped:

Successfully installed VIB in 10.xx.xx.xx 

Functions and scenarios of the script

The SAP HANA HCI health check Python script is executed every 5 minutes, if you want to shorten the time to 1 min, modify the value of RUN_PYC_INTERVAL at the beginning of the bash script.

For all the VMs on the host, the script gets the following info from each individual VM:

vsan.enabled = True/False

sds.solution = ESXI/vSAN_Version

vm_on_vsan = True/False

Compare those values above with the values extracted from the ESXi, then update the values on the VM if necessary.

Testing of the script

  1. Install an SAP HANA supported Linux version based VM on one of your host servers.
  2. Ensure that the latest Open VMware guest tools are installed on your test VM.
  3. Enable host and VM monitor data exchange as documented in SAP note 1606643.
  4. Download and copy the latest sapsysinfo script (<= revision 114), see SAP note 618104 for details and execute it on your test VM.
  5. Example output (search for vSAN in the document):
    1. -------------------------------------------------------------------------------
    2. VMware vSAN info:
    3.  
    4. guestinfo.vsan.enabled = 
    5. guestinfo.SDS.solution = 
    6. guestinfo.vm_on_vsan = 
    7.  
    8. -------------------------------------------------------------------------------
    9. Linux system information:
    10.  
    11. -------------------------------------------------------------------------------
  6. If you do not get the above outputs, check the prerequisites and ensure that the used hypervisor is a VMware hypervisor and the used SDS solution is vSAN.

If a VM is moved to vSAN (or have a newly created vmdk on vSAN), or moved out from vSAN to other datastores, the guestinfo will be updated within 5 mins.

If the VM is storage vMotioned to another host that does not have the script running, you need to manually delete the three variables from the VM configuration file to ensure consistency. You can check inside vCenter if these variables are set for a VM. See Figure 12 for details:

Figure 12. SAP HCI vSAN Specific Configuration Parameters

Updating the script

To update the vib to a higher version, replace the vib file within your Linux/MacOS directory and run the following commands:

[w3-dbc302:hana_vib]$./install_vib_script_remotely.sh -u

Please specify the hosts(Separate by whitespace):

10.xx.xx.xx

login as: root

Password:

 

Validating connectivity to 10.xx.xx.xx

10.xx.xx.xx connectivity validated

 

Uploading setInfo_v1.3.5.vib to 10.xx.xx.xx

Successfully uploaded setInfo_v1.3.5.vib to 10.xx.xx.xx:/tmp

 

Updating VIB in 10.xx.xx.xx

Installation Result:

   Message: Operation finished successfully.

   Reboot Required: false

   VIBs Installed: VMware_bootbank_setInfo_1.3.5-6.7.0

   VIBs Removed: VMware_bootbank_setInfo_1.3.0-6.7.0

   VIBs Skipped:

Successfully update VIB in 10.xx.xx.xx

Uninstalling the script

To uninstall the script from ESXi, run:

[w3-dbc302:hana_vib]$./install_vib_script_remotely.sh -r

Please specify the hosts(Separate by whitespace):

10.xx.xx.xx

login as: root

Password:

 

Validating connectivity to 10.xx.xx.xx

10.xx.xx.xx connectivity validated

 

Removing VIB in 10.xx.xx.xx

Removal Result:

   Message: Operation finished successfully.

   Reboot Required: false

   VIBs Installed:

   VIBs Removed: VMware_bootbank_setInfo_1.3.5-6.7.0

   VIBs Skipped:

Successfully cleared up VMs info, removed vib from 10.xx.xx.xx

SAP HANA HCI on vSAN—Support and Process

Support Status

vSAN is supported as part of an SAP HANA HCI vendor-specific solution, as described in this document. See SAP Note 2718982

The SAP HANA HCI solution vendor is the primary support contact for the customer with escalation possibilities to VMware and SAP. The support of an SAP HANA HCI solution is comparable and very similar to the SAP HANA Appliance support model.

The SAP HANA HCI solution vendor must define the overall usable storage capacity and performance available for SAP HANA VMs. The vendor also defines how many SAP HANA VMs per host are supported and how many vSAN cluster nodes are supported by the vendor.

For supported SAP HANA HCI solutions based on VMware products, check out SAP HANA Certified HCI Solution.

Support Process

If you want to escalate an issue with your SAP HANA HCI solution, work directly with your HCI vendor and follow the defined and agreed support process, which normally starts by opening a support ticket within the SAP support tools.

vSAN Support Insight

vSAN Support Insight is a next-generation platform for analytics of health, configuration, and performance telemetry. Its purpose is to help vSAN users maintain reliable and consistent computing, storage, and network environment and can be used, in addition to the HCI vendor specific support contract, with an active support contract running vSphere 6.5 U1d or later.

How does it work?

See How your Experiences Improve Products and Services for details. 

Conclusion

With vSAN, VMware equips its hardware and OEM partners to build SAP HANA ready HCI solutions. Not only compute, memory, and network, but also internal storage components can be leveraged from a standard x86 server, which dramatically changes the SAP HANA infrastructure economics and the deployment methodology. With SAP HANA HCI powered by VMware solutions, an SAP hardware partner vendor can build true turnkey SAP HANA infrastructure solutions offering customers to deploy SAP HANA in days instead of weeks or months.

Appendix

Best Practices of Virtualized SAP HANA for Skylake Based Server Host Systems

Optimizing SAP HANA on vSphere 6.5 and later - Configuration Parameter List

VMware vSphere can run a single large or multiple smaller SAP HANA virtual machines on a single physical host. This section describes how to optimally configure a VMware virtualized SAP HANA environment. These parameters are valid for SAP HANA VMs running in a standard vSphere environment or SAP HANA on vSAN based SAP HANA HCI configurations and are valid for 2-, 4- and 8-socket hosts.

The listed parameters settings are the recommended settings for the physical server, the vSphere ESXi host, the VM, and the Linux OS to achieve optimal operational readiness and stable performance for SAP HANA on vSphere. These parameter settings are the default settings that should be configured for SAP HANA on vSphere.

The settings described in the section “low latency settings” should only be applied in rare situations where SAP HANA must perform with the lowest latency possible. 

The shown parameters are the best practice configuration parameters and in case of an escalation, the VMware or HCI vendor SAP support engineers will verify these settings.

Physical HOST BIOSParameter Settings

DESCRIPTION

UEFI BIOS HOST 

It is recommended to use (U)EFI BIOS as the standard BIOS version for physical vSphere hosts. All SAP HANA appliance server configurations leverage (U)EFI as the standard BIOS. vSphere fully supports EFI since version 5.0.

Enable Intel VT technology

Enable all BIOS virtualization technology settings.

Configure RAM hemisphere mode

Distribute DIMM modules in a way to achieve best performance (hemisphere mode), use the fastest DIMM modules available for the selected RAM size. Beware of the CPU specific optimal RAM configurations that depend on the available memory channels per CPU.

CPU - Populate all available CPU sockets, use a fully meshed QPI NUMA architecture

To avoid timer synchronization issues, use a multi-socket server that ensures NUMA node timer synchronization. NUMA systems that do not run synchronization will need to synchronize the timers in the hypervisor layer, which can impact performance. See Timekeeping in VMware Virtual Machines for reference.

Select only SAP HANA vSphere supported CPUs. Verify the support status with SAP note 2652670—SAP HANA VM on VMware vSphere.

Enable CPU Intel Turbo Boost

Allow Intel automatic CPU core overclocking technology (P-States). 

Disable QPI Power Management

Static high power for QPI links.

Enable hyperthreading

Double the logical CPU cores to allow ESXi to take advantage for the more available CPU threads. Enable HT regardless the Intel L1 Terminal Fault issue, but make sure to install the latest VMware vSphere patches for this issue.

Enable execute disable feature

Enable the Data Execution Prevention bit (NX-bit), required for vMotion.

Disable node interleaving

Disable node interleaving in BIOS.

Disable C1E Halt state

Disable Enhanced C-States in BIOS.

Set Power Management to High Performance

Do not use any power management features on the server like C-States. Configure in the BIOS static high performance

Set correct PMem mode as specified by the HW vendor for either app-direct or memory mode

Follow the vendor documentation and enable PMem for the usage with ESXi. Note, only app direct memory mode is supported with production level VMs.

Memory mode is only supported with a ratio of 1:4. SAP provides as of today only no-production workload support.

VMware vSAN does not have support for “App-Direct” mode (source) as cache or as a capacity tier device of vSAN. However, vSAN will work with vSphere hosts equipped with Intel Optane PMem in App-Direct mode and SAP HANA VM’s can leverage PMEN according to SAP Note 2913410. Please note that the VMware HA restriction (as described in note 2913410) needs to be considered.

Disable all unused BIOS features, such as:

Video BIOS Shadowable, Video RAM Cacheable, on-board audio, on-board modem, on-board serial ports, on-board parallel ports, on-board game port, floppy drive, CD-ROM, USB.

 

VSPHERE ESXi HOSTParameter Settings

DESCRIPTION

Networking

Use: Virtual Distributed Switches (vDS) to connect all hosts that work together. Define the port groups that are dedicated to SAP HANA, management, and vMotion traffic. Use at least the dedicated 10GbE for vMotion and the SAP App server or replication networks. At least 25 GbE for vMotion for SAP HANA system >= 2 TB is recommended.

Storage

Configuration

If you want to use vSAN, select one of the certified SAP HANA vSAN based HCI solutions and refer to the storage section of this document.

When creating your storage disks for SAP HANA on the VM/OS level, ensure that you can maintain the SAP specified TDI storage KPIs for data and log. Use the storage layout and LVM configuration as a template.

SAP Monitoring

Enable SAP monitoring on the host -> Misc.GuestLibAllowHostInfo = “1” Details: SAP note 1409604.

Without this parameter, no host performance relevant data will be viewable inside an SAP monitoring enabled VM.

SAP HANA VIRTUAL MACHINE Parameter Settings

DESCRIPTION

Tips how to edit the .vmx file

Review tips for editing a .vmx file in VMware KB 1714.

UEFI BIOS Guest 

It is recommended to use (U)EFI BIOS as the standard BIOS version for vSphere hosts and guests. Only with EFI features like Secure Boot are possible. See VMware DOC-28494 for details.  You can configure this with the vSphere client by choosing EFI boot mode. 

If you are using vSphere 6.0 and you only see 2 TB memory in the guest, upgrade to the latest ESXi 6.0 version.

SAP monitoring

Enable SAP monitoring inside the SAP (HANA) VM with the advanced VM configuration parameter tools.guestlib.enableHostInfo = "TRUE".

Details: SAP note 1409604.

Beside setting this parameter, the VMware guest tools need to be installed. See VMware KB 1014294 for details.

vCPU hotplug

Ensure that the vCPU hotplug is deactivated; otherwise, vNUMA is disabled and SAP HANA will have a negative performance impact. Details: VMware KB 2040375.

Memory reservations

Set fix memory reservations for SAP HANA VMs. Do not overcommit memory resources.

It is required to reserve memory for the ESXi host. It is recommended to reserve, depending on the amount of CPU sockets memory for ESXi. 

Typical memory reservations for a host is between 32-64 GB for a 2-socket server, 64-128 GB for a 4-socket server and 128-256 GB for an 8-socket server. These are not absolute figures, since the memory need of ESXi depends strongly on the actual HW, ESXi, and VM configuration and enabled ESXi features like vSAN.

CPU 

Do not overcommit CPU resources and configure the dedicated CPU resources per SAP HANA VM.

You can use hyperthreads when you configure the virtual machine to gain the additional performance. For CPU generations earlier than Cascade Lake, you should consider disabling hyperthreading due to the Intel Vulnerability Foreshadow L1 Terminal Fault. See KB55636 for details. 

You must configure 2 x the cores per CPU socket of a VM for using hypterthreads. For example, 2-socket wide VM on a 28 core Cascade Lake system will require 112 vCPUs.

vNUMA nodes