Understanding Persistent Memory (PMem) in vSphere
Persistent memory is also known as PMem, Non-Volatile Memory (NVM) and Data Center Persistent Memory Module (DCPMM) in the case of Intel’s solution. Persistent Memory can also be presented in different modes such as App Direct Mode, Memory Mode, and Mixed Mode. In addition, PMem can be accessible to a VM as vPMem or vPMemDisk within vSphere. All these different names and modes can be confusing and understanding persistent memory may be a difficult concept to grasp.
So, let’s start with the Basics.
What is Persistent Memory?
Although PMem has been around for a few years, it is arguably still an emerging technology. PMem takes non-volatile media and places it onto a Memory Module (DIMM), and it is installed alongside volatile DIMMs within a server. Since PMem is non-volatile, it is capable of retaining data after any planned or un-planned power events. In essence, Persistent memory fills the gap between DRAM and traditional storage when it comes to performance and cost, by bringing the data closer to the CPU.
Applications that are sensitive to downtime and require high performance, can take advantage of PMem. It is important to highlight that PMem is on the memory bus, thus keeping that data closer to the CPU compared to traditional storage, resulting in lower latency and better performance in most cases. To learn more about PMem High Availability refer to the PMem HA Blog post.
There are a couple different PMem solutions available in the market:
- NVDIMM-N – This solution contains a battery backed, NAND-flash module within a DIMM and data is transferred between the two during startup, shutdown, and power loss.
- Intel Optane DC Persistent Memory – Also known as DCPMM, this module is designed to improve overall performance at near memory speeds. Although the latency of DCPMM is higher than DRAM it is still in the nanoseconds range. One of the attractive aspects of DCPMM is the size of the DDR4 socket compatible module, which currently comes in sizes of 128, 256, and 512 gigabytes per module.
- This offering by Intel has three modes of operations previously mentioned:
- App Direct Mode: Used as low latency persistent memory
- Memory Mode: Used as DRAM
- Mixed Mode: Combination of App Direct and Memory Mode
- This offering by Intel has three modes of operations previously mentioned:
Operation Modes (DCPMM)
With Intel’s Optane DC Persistent Memory, not to be confused with Intel’s Optane SSD, the mode of operation (also called “goal”) is configured within the BIOS and applied on next reboot. During this step, the mode can be set to App Direct Mode, Memory Mode or Mixed Mode.
App Direct Mode
In this mode, the PMem capacity is presented as byte addressable persistent storage to the OS. In the case of vSphere, VMs contained within a host that has PMem modules installed in this mode will allow guests to consume persistent memory. This mode can greatly increase application performance, but some application tuning will likely be required for optimal performance. One example is MSSQL tail-of-the-log (TOFL) tweaks. Typically, an application can benefit from reduced I/O bottlenecks by having access to non-volatile persistent memory in this mode. Some use cases that resulted in performance gains include SAP HANA, Oracle, MSSQL, and REDIS in-memory database. See Intel, and VMware performance whitepapers for more information.
Memory Mode
In memory mode, PMem is exposed to vSphere as DRAM – the PMem module cannot be used as persistent storage in this configuration. Memory mode allows for cheaper, larger capacity PMem modules to be used as main volatile DRAM. Memory mode expands the host’s DRAM capacity to be equal to the amount of PMem installed on the machine, and allows VMs to use this PMem as if it were regular volatile memory. This configuration requires no application modifications to take advantage of the cost and capacity benefits of PMem. Typically, unmodified aaplications in need of more volatile memory can take advantage of this mode.
Mixed Mode
This mode allows users to “carve” a section of the DCPMM module into a percentage dedicate to Memory Mode and the rest to be used as App Direct Mode. For example, configuring DCPMM with Memory Mode 65%, will leave 35% in App Direct mode.
For different use cases, Intel suggests testing the performance on both modes as not all applications have the same I/O profile.
PMem in vSphere
Persistent Memory is supported in vSphere starting with vSphere 6.7 and beyond. Although PMem is not supported on vSAN, VMs on vSAN clusters can still be configured with vPMem and vPMemDisks. For VMs within vSphere to use virtual PMem devices, PMem must be exposed in App Direct mode. vPmem and vPMemDisk are virtual devices that are allocated from the App Direct PMem available on the host.
PMem can be added as vPMem by adding a vNVDIMM device during VM creation or editing. When a vNVDIMM device is added, its size should be equal or less than the total amount of available PMem on the residing host. During this step, an NVDIMM controller is created automatically for this VM to manage the new NVDIMM device. This will look to the guest as if it were running on a host with all of its PMem configured in App Direct Mode.
For the case where we want to present persistent memory as a fast storage device to a VM, a new hard disk can be added to the virtual machine as vPMemDisk. If the host is configured in App Direct Mode, you can configure a vPMemDisk device by modifying the VM storage policy. After adding the hard disk, select the policy named “Host-local PMem Default Storage Policy”. The contents of the new hard disk will be placed on the PMem module.
Summary
In summary, vSphere allows for the integration of emerging technologies such as Persistent Memory and in turn aid in increased performance, lower TCO by adding cheaper memory options at larger capacities, as well as reduced latency on applications.
Additional PMem resources:
Enhancements to Working with PMEM in the vSphere Client
Migrating and Cloning VMs using PMem in vSphere