Protecting vSphere From Specialized Malware
Introduction
Over the past few years, threat actors have adapted their tactics to focus more on specific operating systems and operating environments that carry the most sensitive data, or where an attack can have the greatest effect. By increasing the development and use of specialized techniques, cyber criminals increase their opportunities to steal intellectual property, ransom and extort their victims, and extort victims' customers. Malicious actors gain access to these target environments through a variety of methods, often focused on operational security weaknesses in credential management, network security practices, and unhardened operating environments.
Mandiant has brought to our attention a new variant of malware targeting vSphere, which was discovered in an environment where threat actors may have used operational security weaknesses to compromise a mutual customer. We would like to thank Mandiant for sharing the findings, and we have worked closely with them to quickly arm customers with the right information. Mandiant found no evidence that a vulnerability in a VMware product was exploited to gain access to ESXi during their investigations. They have named the malware artifacts as VirtualPITA (ESXi & Linux), VirtualPIE (ESXi), and VirtualGATE (Windows).
This malware differs in that it supports remaining both persistent and covert, which is consistent with the goals of larger threat actors and APT groups who target strategic institutions with the intention of dwelling undetected for some time. This contrasts with other threat actors and their toolkits who conduct “noisy,” financially-motivated attacks using ransomware. Based on the indications that this new malware was deployed post-compromise, our guidance provides both specific detection and mitigation techniques as well as preventative techniques for strengthening operational security, secure configuration practices, and defense-in-depth.
Mitigation & Detection
VMware has developed mitigation and detection guidance specifically for the techniques outlined in the Mandiant report. You can find this guidance in VMware Knowledge Base Article 89619. The Knowledge Base article also includes a detection script to automate the process of auditing an environment.
Preventative Secure Configuration Practices
VMware practices operational security, secure configuration practices and design, and defense-in-depth when deploying environments. We advocate for our customers to do the same. Here are five ways that customers can address their configurations and processes to better protect themselves from the threats identified in Mandiant's report.
Authentication & Authorization
Attackers know that VMware vSphere is where organizations run their most important workloads. They also know that most organizations like to manage their authentication and authorization with centralized tools and identity providers. Once an attacker has gained access to an organization, their primary target becomes those centralized identity services. For example, if an attacker gains Domain Admin rights on a central Microsoft Active Directory implementation they can add themselves to the groups used for authorization, and simply log in to infrastructure systems as administrators. They may also be able to log into workloads with those same credentials.
Our paper on Practical Ransomware Resilience with VMware vSphere & VMware Cloud Infrastructure (linked below) discusses several ways infrastructure can be configured defensively against these types of attacks, including separation of authorization and authentication, isolation of infrastructure identity sources, use of multifactor authentication, auditing for login successes as well as failures, use of dedicated administration logins, limiting access to management interfaces, and so on.
These techniques should also be implemented for other infrastructure systems like network firewalls, storage arrays, and more. Network firewalls provide defense-in-depth and monitoring capabilities to detect intrusions but not when attackers also have administrative access to disable them.
Secure Boot, Trusted Platform Modules, and Host Attestation
Secure Boot enables ESXi to validate software, drivers, and so on using cryptographic methods at system boot time, determining if those components are legitimate. Support for Trusted Platform Module (TPM) 2.0 in vSphere builds on ESXi Secure Boot by enabling vCenter Server to attest, or validate, the state of the environment by examining data from Secure Boot, as well as system configuration information. vSphere Trust Authority, introduced in vSphere 7, further ties access to encryption keys used for vSAN and VM Encryption to continuous host attestation. Hosts that fail attestation under vSphere Trust Authority are denied access to secrets.
Host Acceptance Levels and VIB Validation
Secure Boot, Host Attestation, TPM, and vSphere Trust Authority build on the cryptographic signing features built into vSphere. The Host Acceptance Level defines the origin of the software that can be installed on ESXi, whether it is from VMware, a partner, or from the community. Software packages used by ESXi are known as VIBs, short for “vSphere Installable Bundles.” Community VIBs are unsigned.
When the Host Acceptance Level is configured to a setting other than “CommunitySupported” it will require cryptographically signed VIB packages to determine the origin of the package. vSphere will check those signatures as part of the Secure Boot and Host Attestation processes, as well as the installation process. This helps ensure malware does not persist as part of the boot process.
When Secure Boot is enabled the use of the “CommunitySupported” acceptance level will be blocked, preventing attackers from installing unsigned and improperly signed VIBs (even with the --force parameter as noted in the report). vSphere 8 takes another step and prevents the execution of unsigned binaries, or binaries installed through means other than a properly signed VIB. Efforts to disable that feature by attackers generates undismissable ESXi alarms as clues that something is happening in an environment.
The vSphere Security Configuration Guides, linked below, are the reference baselines for hardening vSphere environments. Those guides advocate for the use of all these technologies and offer reasonable configurations suggestions, as well as methods for automating their configuration.
Patching & Lifecycle Management
Patching operating systems can be one of the most thankless jobs in IT, but it is the only way to remove a vulnerability from an environment. The ability to migrate running workloads between physical ESXi systems was introduced in 2005 as vMotion, and it revolutionized the ability to keep infrastructure up-to-date independently of workloads. vSphere 7 and 8 each make considerable improvements to vMotion, making migration of workloads seamless again even for large virtual machines. Similarly, vSphere Enhanced vMotion Compatibility (EVC) makes vMotion possible across different generations of CPU families, enabling flexibility and futureproofing.
vSphere 7 also introduced vSphere Lifecycle Manager, which has a declarative way of managing ESXi host configurations. This includes software configuration, where the introduction of an unauthorized VIB would cause the host to become non-compliant with its configuration baseline. Routine patching enables vSphere Administrators to observe these clues, and restarting ESXi enables Secure Boot to reverify the system configuration to detect malware.
Workload Hardening
Virtual machines have always had communication channels available to them between vSphere and the guest operating system (VMCI). These communication channels are used for exchange of information within the VMware product ecosystem and automation of tasks during workload deployment, disaster recovery failover, and so on. The channels are enabled through the VMware Tools and can be disabled granularly as desired.
Most importantly, they require authentication to the guest OS using guest OS credentials. For example, the Invoke-VMScript PowerCLI method to use these channels to run a command on a guest OS requires a username and password for the guest OS. Of course, if workloads are connected to the same compromised identity source then the attackers have access to the workloads, too.
The vSphere Security Configuration Guides, as mentioned above, contain guidance for disabling the host-to-guest communication channels from the workload itself (VMware Tools configuration parameters). As with the other facets of this document, an attacker with privileged access to an identity source used by the workloads will be able to re-enable these channels.
Use of VMware technologies such as VMware Carbon Black Endpoint, the VMware NSX suite of tools, and tools like vRealize Log Insight and vRealize Network Insight harden workloads and offer opportunities to detect and contain attacks in progress.
Conclusion
Every environment is different, so the specifics of what works for you will be different than other organizations. However, implementing the security features discussed above (Secure Boot, Trusted Platform Modules, Host Acceptance Levels, and so on), carefully securing and managing authentication and authorization, implementing access control within your organizations, and keeping environments up to date go a long way towards making life harder for attackers.
Links
- Mandiant Research on ESXi Hypervisor Malware Persistence (original document from Mandiant)
- Protecting vSphere From Specialized Malware (this document, VMware's response to the Mandiant information)
- VMware Knowledge Base 89619 - Mitigation and Threat Hunting Guidance for Unsigned vSphere Installation Bundles (VIBs) in ESXi (including a script to audit ESXi hosts)
- vSphere Security Configuration Guides (baseline hardening guidance for VMware vSphere)
- Mandiant ESXi Detection and Hardening Guidance (Mandiant's document on detecting these issues and hardening an environment)
- Tips for Patching VMware vSphere (practical advice for ensuring patching success, and many ideas here apply to other components as well)
- VMware Ransomware Resource Center (guidance on tactics to help prevent, deter, and recover from attacks)
- VMware Security Advisory Mailing List (please subscribe for proactive notifications of security advisories)
Q&A
This list of questions and answers will be updated as new questions are asked. Please check back.
Who is affected by this?
This is not a security advisory, it is a response to the publication of research done by Mandiant into new malware techniques. All VMware customers running ESXi should consider reviewing the information here, auditing their environments, and applying current defensive security techniques.
When do I need to act?
This is not a security advisory, it is a response to the publication of research done by Mandiant into new malware techniques and is informational. When to act is up to you and will depend on the context of your environment. Consider that threat actors already know this information, though.
What is a VIB?
VIB stands for "vSphere Installable Bundle" and is a software packaging format used by VMware ESXi.
Is this a vulnerability in VMware products?
No. Mandiant found no evidence that a vulnerability in a VMware product was exploited to gain access to ESXi during their investigations. We recommend customers subscribe to the VMware Security Advisory Mailing List (link above) to be notified proactively when security advisories are published.
Will a patch be issued for VMware products?
This is not a vulnerability in VMware products so there will not be a software patch for this issue. VMware Knowledge Base article 89619 contains information on detecting and mitigating issues with unsigned VIBs.
What can I do to protect my environment from this type of malware?
In addition to the aforementioned operational security best practices and the practices documented in the VMware Security Configuration Guide, VMware has provided mitigation guidance specifically for this newly discovered technique in Knowledge Base Article 89619, linked above.
How can I determine if my ESXi hosts are running unsigned VIBs?
Knowledge Base Article 89619 (linked above) provides threat hunting guidance on how to audit your ESXi hosts for unsigned VIBs.
Why isn’t there a CVE number is assigned to this issue?
CVEs (short for Common Vulnerabilities and Exposures) are numerical identifiers assigned to vulnerabilities. As noted above, this disclosure does not involve a vulnerability in VMware ESXi. It is newly discovered malware that is specialized to work in that operating system. The threat actor could have used any number of ways to gain initial access and deploy the malware. Therefore it has not been assigned a CVE number.
What is the severity of this issue?
VMware only assigns severity ratings to vulnerabilities. This issue is not a vulnerability, so does not have a specified severity rating. We do urge customers to design, implement, and operate their systems using security principles and features that help prevent these types of attacks.
Do these issues have other names?
Mandiant has named these issues VirtualPITA, VirtualPIE, and VirtualGATE.
Are VMware Cloud and hosted products affected?
VMware is always evaluating new security information to determine relevance to all of our products, including VMware Cloud and hosted services. If issues are found the VMware SREs, working around the clock, take action to mitigate it. VMware delivers relevant information to customers as a message inside hosted, cloud, and software-as-a-service products. We do it this way because hosted products are implementations of the products, and, as all of our on-premises customers know from experience, have different operational considerations than just a software release. Please check the administrative consoles of those services for relevant messages about this VMSA. For questions about the service please follow the support processes for that service. Thank you!
How do I configure sshfs and Yara to scan ESXi hosts?
The suggestion of sshfs and Yara was made by Mandiant. Those tools, while powerful, are not supported by VMware. In general VMware does not recommend enabling SSH on vSphere, but if customers wish to do it they can do it in bulk with a PowerCLI command:
Get-VMHostService -VMHost * | where {$_.Key -eq 'TSM-SSH'} | Start-VMHostService
Take care to disable SSH again as soon as possible using Stop-VMHostService.
I am using a third-party solution, like Dell EMC VxRail, HPE SimpliVity, and so on. Is it safe for me to apply these scanning techniques?
VMware is proud of the robust partner ecosystem and community built around our products, but we cannot speak to our partners’ solutions. Nor would they want us to.
Engineered and integrated solutions like Dell EMC VxRail, HPE SimpliVity, and even VMware Cloud Foundation control their patch levels and configurations as part of their qualification and testing processes. Using security guidance that is not explicitly for that product and product version is never advised. If you have additional engineered and integrated solutions in use, you should contact those vendors directly for guidance.
My scanning has found unsigned VIBs. Are we compromised?
The Knowledge Base article discusses this issue. VIB signing became prominent in vSphere 6.5, but hosts that may have been upgraded for several major versions might still have unsigned packages left over, as Update Manager did not actively clean those up. Similarly, parts of the greater VMware third-party product ecosystem may have distributed unsigned VIBs in the past, which may still be present. vSphere Lifecycle Manager’s “single image mode” in vSphere 7 and 8 takes a declarative stance to host configuration and will clean up unnecessary packages.
Please also ensure that you are scanning for unsigned VIBs using the scripts and techniques in the Knowledge Base article. Non-VMware guidance for scanning signatures may not return accurate data.
How do I scan my ESXi logs for this?
Your log management/event management/SIEM/Log Insight systems should allow you to search for log entries containing "esxcli" and "force." We do not suggest using local ESXi logs as attackers with privileges to install software also will have the ability to alter local logs to remove evidence of their activity.
Change Log
All times are in Pacific Daylight (-0700)
2022-09-29, 0600: Initial release.
2022-09-30: 1100: Additional Q&A, added the KB link near the top so it is easier to find.