Key Rotation Options for vSAN ESA in VMware Cloud Foundation 5.1 and vSAN 8 U2
Storage platforms that power environments must meet a variety of customer requirements. Since the initial announcement of the Express Storage Architecture (ESA) in vSAN 8, and subsequent improvements in vSAN 8 U1 and U2, we've been able to deliver all new levels of performance, resilience, and manageability that makes it more capable than ever.
But we cannot forget about security. For vSAN 8 U2, which is also a part of VMware Cloud Foundation 5.1, we’ve improved the key rotation options within the ESA to provide more flexibility in complying with security standards. Let's look at what has changed.
Multiple Key Rotation Options for ESA
Key management is at the heart of any type of encryption offering. In the Original Storage Architecture (OSA), two key rotation methods were available to meet specific objectives by security teams in an organization.
- Shallow Rekey: An existing Key Encryption Key (KEK), which is managed by the key provider, such as a KMS solution, or the vSphere Native Key Provider (NKP), is recreated. All Disk Encryption Keys (DEK)s remain the same but are re-wrapped with a new KEK. Shallow rekey operations are very fast due to little to no data movement, and are the most common of the two key rotation methods.
- Deep Rekey: The existing KEK and DEKs are recreated, and the data is re-encrypted using the new DEK. This type of operation incurs significant data movement as it must rewrite the existing data again using the new keys. As a result, it is not as common as a shallow rekey, and sometimes not even available in a storage solution that offers encryption.
While the OSA supported both shallow and deep rekey operations, the initial version of vSAN ESA only supported a shallow rekey capability. With vSAN 8 U2, both shallow rekey and deep rekey operations are available with clusters running the ESA.
A deep rekey will perform two basic steps against the data at rest. First, the storage device will undergo a disk format conversion (DFC), where each storage device is evacuated and reclaimed using a new key. Then, the objects residing on the devices will undergo an object format conversion (OFC). The object is decrypted by the old key and re-encrypted by the new key.
Figure 1. Deep rekey process for the ESA in vSAN 8 U2, included in VMware Cloud Foundation 5.1.
A deep rekey operation is a hands-off automated process that takes just a few clicks in the UI to perform, but since it does introduce data movement across the storage devices in a cluster, it does require some thought and care to ensure the appropriate time and use.
Deep Rekey Considerations
Just as with the OSA, a deep rekey in the ESA introduces data movement, as it cycles through the respective storage devices. While this is still a potentially resource intensive event, it is not as invasive as a deep rekey operation performed in a cluster running the OSA.
- ESA has a much more efficient encryption process. Encryption occurs at the top of the storage stack, which eliminates a significant amount of the CPU and I/O amplification that occurred with encryption processes in the OSA.
- ESA has a smaller boundary of maintenance. With the ESA, storage devices are evacuated and repopulated at the discrete storage device level, compared to the OSA with its use of disk groups. For more information on the benefits of a smaller boundary of maintenance and failure, see the post: "The Impact of a Storage Device Failure in vSAN ESA versus OSA."
To help ensure that a deep rekey operation does not consume an overwhelming amount of resource, the ESA may employ dynamic throttling mechanisms to help preserve resources. For example, it may limit the number of concurrent objects being rekeyed at any given point in time, based on the conditions of the cluster. Under times of resource contention, it may also use the adaptive resync capability of the host's storage stack as well as vSAN ESA's adaptive network traffic shaping feature to ensure that VM traffic is prioritized over resynchronization traffic. These mechanisms are fully automated, and from the perspective of the administrator, does not require any additional effort.
Recommendation: Monitor the performance metrics available in the vSAN performance service during your first deep rekey operation. This will help you determine if, and where you see any impact on guest VM activity.
Note that while ESA in vSAN 8 U2 does support enabling encryption after the initial deployment of a cluster, vSAN ESA does not currently support turning off encryption on a cluster once it is enabled. Turning off and then enabling encryption was sometimes the method used for those wanting to reset encryption keys but were unfamiliar with a deep rekey process.
For more information on encryption capabilities and operational considerations, see the "vSAN Encryption Services" document out on core.vmware.com.
Summary
The Express Storage Architecture in vSAN 8 U2 improves its security stance by allowing multiple key rotation methods. This allows our customers to meet the common key rotation requirements necessary to provide a secure and scalable storage platform for your VMware Cloud Foundation and vSphere environments.