ESXi System Storage When Upgrading

Overview

As you are aware of the fact that, usage of standalone SD card or USB devices are not recommended starting from vSphere 7 Update 3. This means that the system will continue to run with warnings as described here. It is advised that you should have, and in the future, must have a locally attached persistent storage device for storing ESX-OSData partition.

OPTION 1: Long Term (Supported)

System Boot Partition and ESX-OSData Partition on the same high-endurance, locally attached persistent storage device. This should be the preferred configuration for the long term.

OPTION 2: Legacy (Supported)

System Boot partition on SD card or USB device, and ESX-OSData partition on a high-endurance, locally attached persistent storage device. This configuration is also supported.

OPTION 3: Not Recommended (Supported)

Standalone SD card or USB device to store system boot partition and RAMDisk for some portion of ESX-OSData Partition. No high-endurance, locally attached persistent storage device available for ESX-OSData Partition. This configuration is not recommended starting from vSphere 7 Update 3.

Storage Requirements Upgrades

Installing ESXi 7.0 requires a persistent storage device that is a minimum of 32 GB. Upgrading to ESXi 7.0 requires a boot device that is a minimum of 4 GB. When booting from a local disk, SAN or iSCSI LUN, at least a 32 GB disk is required to allow for the creation of system storage volumes, which include a boot partition, boot banks, and a VMFS-L based ESX-OSData volume. The ESX-OSData volume takes on the role of the legacy /scratch partition, locker partition for VMware Tools, and core dump destination. Starting with ESXi 7.0 Update 3, a VMware Tools partition is automatically created on the RAM disk and you see warnings to prevent you create partitions other than the boot bank partitions on flash media devices.

Boot from SAN (FC/FCoE/iSCSI) is supported, requires a dedicated LUN, minimum of 32 GB is recommended

Please refer System Storage Requirement from ESXi 7.x for more details.

Configuration	Status	Notes	Device Considerations
High Quality Boot Device	Supported, Preferred Long Term Support	Consolidation of System boot, bootbank, and ESX-OSData Partition on the same device	Device Endurance: 100TBW Locally attached devices such as NVMe (138 GB minimum) SSD(138 GB minimum) M.2 Industrial Grade (138 GB minimum) HDD (32 GB Minimum) Managed FCoE/iSCSI LUN (32 GB Minimum)
Low Quality Boot Device + High Quality Device	Supported, Legacy Configuration	Bootbank and ESX-OSData partition remain separate. SD card or USB device should only be considered to storage system boot partition	Low-Quality Device SD card or USB drive Minimum 8 GB Device Endurance: Minimum 1 TBW High-Quality Device Device Endurance: 100TBW Locally attached devices such as NVMe (138 GB minimum) SSD(138 GB minimum) M.2 Industrial Grade (138 GB minimum) HDD (32 GB Minimum) Managed FCoE/iSCSI LUN (32 GB Minimum)
Low Quality Boot Device (Only SD card or USB device as the boot media), No High Quality Device	It is not recommended in vSphere 7 Update 3. Still runs but with warnings. Please refer VMware KB Article 85615	All the partitions, including the ESX-OSData partition, are installed on the same low-quality boot media. This is a not recommended configuration.

Partition Layout

To quickly recap what’s in the previous blog post, let’s look at how the partition layout changed between vSphere 6.x and vSphere 7. The small & large core-dump, locker, and scratch disk are consolidated into the new ESX-OSData partition.

Whether you freshly install or upgrade to vSphere 7, the partition layout as shown in the diagram above is applied. This partitioning reflects what happens in the vSphere upgrade process when the ESXi system storage media is HDD or SSD. The (system storage related) upgrade steps are:

Backup potential partner VIBs (kernel modules), contents of the active boot-bank, locker and scratch partitions to memory (RAM).
Cleanup all system partitions, non-datastore partitions are not destroyed.
If the upgrade media does not have an existing VMFS partition, the upgrade process creates a new GPT partition lay-out.
Create partitions (book-banks and ESX-OSData)
Restore the contents from RAM to the appropriate partitions.

Upgrade Scenarios

Follow steps to move away from deprecated configurations involving the usage of standalone SD card or USB device

Upgrading ESXi 6.7 with Standalone SD card or USB Device to ESXi 7.x with an additional disk

Please follow the below steps If there is no persistent storage available for ESXi 6.7 host

Add a high-endurance, locally attached persistent storage device on ESXi 6.x host
Upgrade ESXi Host to ESXi 7.x.
Enable autoPartition=True, This will auto partition the first unused boot device to be used as ESX-OSData partition. Please refer VMware KB Article 77009
This will ensure SD card or USB device storing System boot partition and Newly added storage device is storing ESX-OSData partition, OPTION 2 in above diagram.

Partition newly added storage device to be used as ESX-OSData partition

If the ESXi Host is already upgraded to ESXi 7.x and running with a standalone SD card or USB device

Add a high-endurance, locally attached persistent storage device
Boot the ESXi host and set autoPartition = True, it will auto partition the first unused boot device to be used as ESX-OSData partition. Please refer VMware KB Article 77009
This will ensure SD card or USB device storing System boot partition and Newly added storage device is storing ESX-OSData partition, OPTION 2 in above diagram.

Persistent Storage Warnings

In the scenario of using a 4 GB boot device and no local disk is found, ESXi is running and displays a persistent storage warning. This warning, in summary, is a state where logs and state might not be persistent, with a side effect that it can cause boot up to be slow. If you have an HDD or better media for the OSData partition, then the system won't enter this state. For USB/SD card boot devices, without an extra HDD/SSD/NVMe to create the OSData partition on, ESXi goes into this state because the USB/SD device isn't used for logs or persistent storage.

When in this state, a sysalert is shown: ALERT: No persistent storage available for system logs and data. ESX is operating with limited system storage space, logs and system data will be lost on reboot. Solving this requires adding a local disk or flash device and run the instructions found in KB article 77009.

Move away completely from the usage of SD card or USB device

Add a locally attached persistent storage device.
Re-install ESXi 7.x on a locally attached storage device
Please refer VMware KB Article 2042141 If you want to backup and restore ESXi configuration.
This will ensure that all the partitions are stored on a high-endurance, locally attached storage device.

Conclusion

VMware is moving away from the support of SD cards and USB drives as boot media. ESXi Boot configuration with only SD card or USB drive, without any persistent device, is strongly not recommended with vSphere 7 Update 3. Customers are advised to move away from SD cards or USB drives completely. If that is not currently a feasible situation, please ensure a minimum of 8GB SD cards or USB drive is present and an additional minimum of 32 GB locally attached high endurance device available for ESX-OSData Partition.