ESXi System Storage When Upgrading

Overview

As you are aware of the fact that, usage of standalone SD card or USB devices are deprecated starting from vSphere 7 Update 3. The deprecation means that the system will continue to run with warnings as described here. It is advised that you should have, and in the future, must have a locally attached persistent storage device for storing ESX-OSData partition.

 

image-20210930150044-1

OPTION 1: Long Term (Supported)

            System Boot Partition and ESX-OSData Partition on the same high-endurance, locally attached persistent storage device. This should be the preferred configuration for the long term.

OPTION 2: Legacy (Supported)

            System Boot partition on SD card or USB device, and ESX-OSData partition on a high-endurance, locally attached persistent storage device. This configuration is also supported.

OPTION 3: Deprecated

            Standalone SD card or USB device to store system boot partition and RAMDisk for some portion of ESX-OSData Partition. No high-endurance, locally attached persistent storage device available for ESX-OSData Partition. This configuration is deprecated starting from vSphere 7 Update 3.

 

 

Storage Requirements Upgrades

Installing ESXi 7.0 requires a persistent storage device that is a minimum of 32 GB. Upgrading to ESXi 7.0 requires a boot device that is a minimum of 4 GB. When booting from a local disk, SAN or iSCSI LUN, at least a 32 GB disk is required to allow for the creation of system storage volumes, which include a boot partition, boot banks, and a VMFS-L based ESX-OSData volume. The ESX-OSData volume takes on the role of the legacy /scratch partition, locker partition for VMware Tools, and core dump destination. Starting with ESXi Update 3, a VMware Tools partition is automatically created on the RAM disk and you see warnings to prevent you create partitions other than the boot bank partitions on flash media devices.

Boot from SAN (FC/FCoE/iSCSI) is supported, requires a dedicated LUN, minimum of 32 GB is recommended

Please refer System Storage Requirement from ESXi 7.x for more details.

Configuration

Status

Notes

Device Considerations

High Quality Boot Device

Supported, Preferred Long Term Support

Consolidation of System boot, bootbank, and ESX-OSData Partition on the same device

  • Device Endurance: 100TBW
  • Locally attached devices such as
    • NVMe (128 GB minimum)
    • SSD(128 GB minimum)
    • M.2 Industrial Grade (128 GB minimum)
    • HDD (32 GB Minimum)
    • Managed FCoE/iSCSI LUN (32 GB Minimum)

Low Quality Boot Device + High Quality Device

Supported, Legacy Configuration

Bootbank and ESX-OSData partition remain separate. SD card or USB device should only be considered to storage system boot partition

 

Low-Quality Device

  • SD card or USB drive
  • Minimum 8 GB
  • Device Endurance: Minimum 1 TBW

High-Quality Device

  • Device Endurance: 100TBW
  • Locally attached devices such as
    • NVMe (128 GB minimum)
    • SSD(128 GB minimum)
    • M.2 Industrial Grade (128 GB minimum)
    • HDD (32 GB Minimum)
    • Managed FCoE/iSCSI LUN (32 GB Minimum)

 

Low Quality Boot Device (Only SD card or USB device as the boot media), No High Quality Device

  • It is being deprecated in vSphere 7 Update 3.

 

 

All the partitions, including the ESX-OSData partition, are installed on the same low-quality boot media. This is a degraded mode configuration.

 

 

 

Partition Layout

To quickly recap what’s in the previous blog post, let’s look at how the partition layout changed between vSphere 6.x and vSphere 7. The small & large core-dump, locker, and scratch disk are consolidated into the new ESX-OSData partition.

Whether you freshly install or upgrade to vSphere 7, the partition layout as shown in the diagram above is applied. This partitioning reflects what happens in the vSphere upgrade process when the ESXi system storage media is HDD or SSD. The (system storage related) upgrade steps are:

  1. Backup potential partner VIBs (kernel modules), contents of the active boot-bank, locker and scratch partitions to memory (RAM).
  2. Cleanup all system partitions, non-datastore partitions are not destroyed.
  3. If the upgrade media does not have an existing VMFS partition, the upgrade process creates a new GPT partition lay-out.
  4. Create partitions (book-banks and ESX-OSData)
  5. Restore the contents from RAM to the appropriate partitions.

 

Upgrade Scenarios

Follow steps to move away from deprecated configurations involving the usage of standalone SD card or USB device

Upgrading ESXi 6.7 with Standalone SD card or USB Device to ESXi 7.x with an additional disk

Please follow the below steps If there is no persistent storage available for ESXi 6.7 host

  1. Add a high-endurance, locally attached persistent storage device on ESXi 6.x host
  2. Upgrade ESXi Host to ESXi 7.x.
  3. Enable autoPartition=True, This will auto partition the first unused boot device to be used as ESX-OSData partition. Please refer VMware KB Article 77009
  4. This will ensure SD card or USB device storing System boot partition and Newly added storage device is storing ESX-OSData partition, OPTION 2 in above diagram.

Partition newly added storage device to be used as ESX-OSData partition

If the ESXi Host is already upgraded to ESXi 7.x and running with a standalone SD card or USB device

  1. Add a high-endurance, locally attached persistent storage device
  2. Boot the ESXi host and set autoPartition = True, it will auto partition the first unused boot device to be used as ESX-OSData partition. Please refer VMware KB Article 77009
  3. This will ensure SD card or USB device storing System boot partition and Newly added storage device is storing ESX-OSData partition, OPTION 2 in above diagram.

 

Degraded Mode

In the scenario of using a 4 GB boot device and no local disk is found, ESXi is running in a so-called ‘degraded mode‘. Degraded mode, in summary, is a state where logs and state might not be persistent, with a side effect that it can cause boot up to be slow. If you have an HDD or better media for the OSData partition, then the system won't enter degraded mode. For USB/SD card boot devices, without an extra HDD/SSD/NVMe to create the OSData partition on, ESXi goes into degraded mode because the USB/SD device isn't used for logs or persistent storage.

When in degraded mode, a sysalert is shown:  ALERT: No persistent storage available for system logs and data. ESX is operating with limited system storage space, logs and system data will be lost on reboot. Solving this requires adding a local disk or flash device and run the instructions found in KB article 77009.

Move away completely from the usage of SD card or USB device

  1. Add a locally attached persistent storage device.
  2. Re-install ESXi 7.x on a locally attached storage device
  3. Please refer VMware KB Article 2042141 If you want to backup and restore ESXi configuration.
  4. This will ensure that all the partitions are stored on a high-endurance, locally attached storage device.

 

Conclusion

VMware is moving away from the support of SD cards and USB drives as boot media. ESXi Boot configuration with only SD card or USB drive, without any persistent device, is deprecated with vSphere 7 Update 3. In future vSphere releases, it will be an unsupported configuration. Customers are advised to move away from SD cards or USB drives completely. If that is not currently a feasible situation, please ensure a minimum of 8GB SD cards or USB drive is present and an additional minimum of 32 GB locally attached high endurance device available for ESX-OSData Partition.

Important Links

Filter Tags

Storage Upgrade ESXi 7 vSphere 7 ESXi Document Best Practice Technical Overview Intermediate Design Deploy