Patching Overview, Including Critical Security Patches

VMware Cloud Foundation allows for a streamlined patching and upgrading experience, however I get a number of questions on what that means, and does it slow down my ability to move fast for critical patches.  Let’s start on this topic with covering how patching works and how VMware releases software bundles.

If we take a look at how patching has been done traditionally you have a dependency mapping effort that goes into applying updates in the correct order to ensure version compatibility.  In traditional three tier environments it looks something like this:

There are many layers to this patching cycle, for example you have your traditional vSphere environment, storage array(s), fiber channel switch, server vendor, and finally your operations/monitoring stack.  In my experience with customers this becomes a complex entangled web that essentially needs a project plan assembled and it may take three to six months before it can be applied to the platform.  Also, the baseline that you are starting with may not be a consistent set of ESXi hosts.  Working through some of these projects in my career I have seen upwards of 20 different ESXi versions, and uptimes of five years plus.  Now that sure speaks volumes about the stability of the platform, but my security sense says that it is most likely vulnerable.

Enter Cloud Foundation and its ability to lifecycle manage the software it deploys. This feature makes patching  a full Software Defined Data Center easy.  Reducing the complexity of patching operations and the resources required to perform them empowers businesses to update frequently.  In turn, the environment becomes more stable and more secure through the application fo the patches.  The foundation of the lifecycle management feature begins with the Cloud Foundation Bill of Materials (BOM). This defines the various products and product versions that are contained in a given release of Cloud Foundation. The BOM is the result of collaboration between various groups within VMware, such as engineering, product management, and support. Factors such as architecture are also considered when producing a BOM.  The result is a BOM that has compatibility across each component and has been engineered to work together.  For example, here is a link to our BOM page showing all the components at each shipping version of VCF (KB).  As a customer, being provided a software BOM like in Cloud Foundation, you can ensure full compatibility up and down the stack in addition to a stable platform.

A software BOM and validated deployment is great, but over time updates need to be applied.  VCF allows you to upgrade from your current version to a target version, with the assurance that dependency mapping has been certified by our engineering teams and the upgrade path has been validated and is released in the form of a bundle.  A bundle is a release mechanism used for VCF that includes the update package, descriptor file, and a checksum file.  The descriptor file is where dependency mapping happens to ensure that the validated upgrade path is ordered properly. Based on our validated upgrade path, bundles become applicable in the exact order of upgrade needed to ensure a safe transition.  From an SDDC Manager perspective you will see the following bundle types listed for download:

1. VMware Update Bundles
2. VMware Software Install Bundles
3. Third Party Updates

VMware Update Bundles are one of the most common types of bundles that will download in a VCF environment.  These are the bundles that upgrade you from version x to version y, or they may be a security patch (more on that soon).  These bundles will be released on a regular basis and will apply to the following components in a VCF environment:

• Cloud Foundation (SDDC Manager, VMware Cloud Foundation services, and drift remediation)
• vSphere (vCenter Server and ESXi)
• NSX
• vRealize Suite

VMware Software Install Bundles are like the name implies new software installation.  After upgrading your environment from version x to version y, you would want to deploy new WLD’s or Clusters at the latest version that your system is running.  Therefore, these will generally be download for any major, minor, and maintenance release, but will not necessarily be downloaded for a patch release.  Also, for net new deployments of VCF binaries are usually only available down to the maintenance release.  VCF Release versions are categorized in the following way:

           Major.Minor.Maintenance.Patch

A major release would be something like 4.0 where a minor release would be 4.1, a maintenance release would be 4.0.1, and a patch release would be something like 4.0.0.1.

Third party updates are the least common and are usually incorporated into the SDDC Manager upgrade itself.  For example, SDDC Manager runs Photon OS, therefore sometimes it makes sense to get Photon OS on the latest version of the Operation System and it may make sense to replace the appliance like we have grown accustomed to with vCenter updates.  The third party portion of this update is a script that runs that migrates the database on the SDDC Manager from the old appliance to the new appliance.  However, when running a patching process for SDDC Manager you may see Third Party listed where they may be restarting services or cleaning up the upgrade process.

Security updates is also a subset of the VMware Update Bundles, however the security updates are not differentiated as its own category within SDDC Manager.  The Engineering, Architecture, and Product Management teams are keeping a pulse on any security updates affecting the core set of software delivered in VCF.  When a critical patch is released for a component of VCF you can be assured we are working cross functionally to confirm full stack compatibility.  To give you an example of what that may look like is as follows:

Therefore, the Cloud Foundation team has committed to the following schedule for security patches:

Just one thing to note on the ‘Greater than 9’ CVSS Score category, VMware is committed to deliver the patches as quickly as possible in this scenario.  In most instances an update has been available within 72 hours, however in some cases it may take up to a one week to perform additional interoperability checks.

Async Patch Tool

There may be situations where a critical security patch or bug fix becomes necessary for a VMware Cloud Foundation instance, even before the fix is available in an update bundle that is part of the normal delivery process described above. In those cases, VMware provides the ability to apply individual product updates without waiting for the full update to be released. This is accomplished through the Async Patch Tool, a command-line tool that VCF administrators run on the SDDC Manager system to import specific patches.

The Async Patch Tool can be used with VCF environments that have Internet access as well as those that do not have a direct connection to the Internet. These cases are known as online and offline modes, respectively.

A list of async patches can be found at VMware KB article 88287 and the product documentation explains how to use the tool to work in both online and offline VCF environments.

For a technical walkthrough of the Async Patch Tool, there are two blog posts available, covering both online and offline cases.

Conclusion

To review, we have covered an overview of VCF patching, bundle types, and finally patching stance for security related fixes.  Additional resources on security related fixes can be found here: VMware Security Advisory, Click on Advisory ID and enter Cloud Foundation.