Encrypt Without Compromise with Native Key Provider and vSAN Encryption
Trading security for usability has often been a common problem in enterprise IT. Data at rest encryption is a powerful tool to prevent data leaks, but operational complexity or costs associated can prevent it from being adopted in all use cases. Introduced in vSphere 6.5 vSAN and VM encryption were introduced, allowing a remote KIMP compliant key manager to provide assurances that data would not easily walk off with the drives in the hosts. A challenge remained, that deploying key managers to all locations (especially small sites) could prove to be excessively complex and expensive.
vSphere 7 Update 2 introduces a Native Key Provider that allows for cost-effective and easy to deploy key management for VSAN and VM encryption. I recently spun this up in the lab and had a few quick observations.
Speed of deployment
After a few clicks in the WebUI and I had created a new key. Pointing a new cluster at this key manager was as simple as picking the vSAN cluster from a dropdown.
Quick Operational Tips:
1. Buy TPMs for your servers, and store the keys in the TPM rather than the boot device. This will prevent theft of anything less than the servers in the cluster + their attached TPMs from resulting in a potential data spill. While it might be easy to walk out of a data center with a few hard drives, It is fairly impossible to walk out with 4 a few hundred pounds of the server under your arm. TPMs are ~$50 at most and should be considered just as essential on new servers as out of band management.
2. Make a backup of your keys (set a password and make sure to put this in a safe location. Given in my lab I”m protecting top-secret family cookie recipes I chose a USB drive in a fire safe.
3. To make a backup of your keys, you need to log in to the VCSA using the FQDN rather than IP address.
4. Once you have manually backed up the keys once, the keys will end up in your VCSA file-level backups.
Chaos in the cluster
When I test out new features I like to try to break them. I mean, just do everything wrong, avoid reading the documentation. So I spun up a cluster and deleted the keys. I’ll note there’s a UI warning that invokes a slider and red confirmation check, so it is hard to fall into sysadmin highway hypnosis and accidentally do this, but I did it anyway.
I quickly checked on my cluster and discovered…. The virtual machines had kept running. As Bob Plankers explains in this great video, the keys are cached onto the hosts.
Looking around the vSAN health alarms had triggered I had clear notification of the damage done.
Rebooting hosts, exposed that the keys were still able to be pulled from TPM, and survive power loss.
Order is restored
While it was impressive to see that the cluster could support the destruction of the keys in the native key provider, and reboot of hosts it was time to put the cluster back to proper health.
To fix the issue, I created a new key in the key manager and then pointed the VSAN cluster at the key. The next step I took was to force a full deep rekey (full re-write of data).
Upon completion, the alarms had cleared and orders had been brought back to my cluster.
Final thoughts
While I don’t advocate anyone goes deleting their keys and rebooting their cluster it was impressive to see how resilient to failure as well as how easy it is to set up vSAN Encryption using the new native key providers. Deployed with data-in-transit encryption, native key providers allows for powerful but easy end-to-end encryption for vSAN clusters.
Additional Reading: