Tips for Patching VMware vSphere & Cloud Infrastructure
Beyond the act of using vSphere Lifecycle Manager (on vSphere 7) or vSphere Update Manager (on vSphere 6.x), these are some techniques that have been used over the years for ensuring success of patching and upgrades. Some of these ideas aren't just technical, they are also people & process improvements, too. Patching is often as much a process issue in organizations as it is a technical task!
- vSphere Administrators understand that patching vCenter Server doesn't impact workloads, and that vMotion will move workloads seamlessly so that ESXi can be patched. However, other people in your organization may not know this. Taking the time to explain this to others, like change managers, risk assessors, and managers, can pay dividends in getting patching approved.
- Ensure that your organization has the vCenter Server Appliance (VCSA) root & firstname.lastname@example.org account passwords stored correctly and are not locked out. By default, the VCSA root account locks itself after 90 days, which may be an unwanted surprise if you need it in an emergency. Prior to patching we suggest verifying these accounts work correctly, recovering the passwords if needed (which sometimes takes a restart of vCenter Server), and then changing them after the work is done as a good security practice.
- Ensure that time settings are correct on the appliance. Many issues on systems can be traced to incorrect time synchronization. The maintainers of the open-source NTP software suggest configuring four NTP servers (but never two – if one is wrong you’ll never know which one!).
- Ensure that there is a properly configured A (forward) and PTR (reverse) record for vCenter Server in DNS. You might think “these are basic and were set up long ago” but it only takes a second to check, and sometimes you learn interesting things. PTR records are required for vCenter Server and omitting them is not an accepted security practice.
- Ensure that vCenter Server’s file-based backup & restore is configured and generating scheduled output. You can configure this through the Virtual Appliance Management Interface (VAMI) on port 5480/tcp on the VCSA.
- Take a snapshot of the VCSA prior to the update, and preferably from the ESXi host client after the VCSA has been shut down gracefully & cleanly. Snapshots have performance impacts, so ensure that you delete it soon after the upgrade is verified.
- An experienced sysadmin once suggested that if it has been a while since a system has been restarted it is often a good idea to simply restart it in place, let it come back up again and prove that it’s working well. Otherwise, you won’t be able to tell whether a problem was pre-existing or because of the work that just happened. An extra reboot does add management interface downtime, but if corrective action is needed it shortens the time to restore the service.
- If vSphere HA has been configured with a custom isolation address (das.isolationaddress) ensure that it is NOT set to the vCenter Server itself, or that multiple addresses are specified (das.isolationaddress0 through das.isolationaddress9) so that one address becoming unavailable does not trigger HA failover.
- Where possible, minimize the number of plugins installed in vCenter Server. Modern zero-trust security architecture practices discourage connecting systems in these ways, to make life harder for attackers (a single pane of glass is nice for cybercriminals, too, especially if it allows access to your backups). Fewer things installed also means fewer things to worry about from a compatibility perspective and makes upgrades and patching much less work.
- Minimize additional installed VIBs on ESXi, and where possible, use "stock" VMware ESXi versions instead of OEM customized ones. This helps avoid issues with VIB version conflicts that can arise from vendor packages. vSphere Lifecycle Manager on vSphere 7 and newer makes it easy to add OEM driver packages and additional OEM software if you truly need it. Remember that, from a security perspective, less is more when it comes to installing extra software. Don't install things you don't absolutely need.
- Use DRS groups & rules to keep vCenter Server on a particular ESXi host so that if there is an issue you can find the VCSA easily using the ESXi host client. Create a host group that has only that one host in it and create a “should” affinity rule to keep that VM on that host. By using “should” you enable the VCSA to be automatically moved by DRS for normal host patching. Ensure that a management workstation can get to the host client interface on that ESXi host, and that you can log in.
- Don't forget about Platform Services Controllers (PSCs)! They are considered part of vCenter Server and all PSCs that replicate together should be updated before you patch vCenter Server. Ensure that NTP, DNS, and all the other considerations above are checked and valid for your PSCs, too.
- vCenter Server should always be updated before ESXi, so the overall order for vSphere is: PSCs, then vCenter Server(s), then ESXi hosts.
- Seeing, or not seeing, weird things after you update vSphere? Clear your browser cache as part of your update process to ensure that the latest vSphere Client components download properly.
For more VMware Security Resources please visit:
About the Author
Bob Plankers has been bridging people, process, and technology in information security and IT infrastructure operations for nearly three decades. Prior to joining VMware he spent 24 years in enterprise IT, wearing many hats (both figuratively and literally) from support to developer to systems engineer to manager, and enjoys bringing his experience to bear on modern issues in IT. Have a suggestion, a comment, or see an error in this document? Please email him at email@example.com.