vSphere's Internal CDN System - VMware Tools Guest Store
VMware Tools version 11.2 is a feature packed release. Apart from general enhancements and updates like:
- The WDDM driver now supports larger cursors drawn by the VMware SVGA device (up to 128x128 pixels from 64x64 pixels) instead of the software. This requires a virtual machine running hardware version 18 or later. For information on the list of products tied with their respective virtual hardware versions, refer to KB 1003746 and KB 1022060.
- Added support to enable WDDM driver distribution on Windows Update.
- Note: WDDM driver will be made available through Windows Update service, when the driver publication process completes.
- Updated openssl version to 1.1.1g.
- Updated libxml2 version to 2.9.10.
- Updated WDDM, vsepflt, and vnetwfp drivers for fixes and feature support.
This blog focuses on the VMware Tools GuestStore, a feature request we received from our customers to have a safe and straightforward mechanism to share content with multiple VMs.
Over the years, VMware has developed and acquired several management solutions (Carbon Black, AppDefense, etc.). While some solutions do not require an agent, many need an agent to be running inside the VM.
Post-deployment distributing updates to these agents have been a long-standing challenge. Today customers employ multiple distribution solutions to solve this problem. For example, Windows Server Update Services (WSUS) and Microsoft System Centre based solutions are used for a windows-based environment. On the other hand, for Open source/Linux based environments, vendor-provided RPM/Debian package repositories are used.
While these solutions serve the purpose, they also come with some inherited shortcomings:
- To leverage these solutions completely, agents need to be packaged in the distribution solution's format.
- Hosting an application in Linux OS vendor repositories requires open sourcing the application, which may not be possible for some agents.
- Customers are required to maintain another service to distribute VMware agents.
- All of these solutions require VM to be on the network as a prerequisite.
So, what is VMware Tools GuestStore, and how does it help?
VMware Tools GuestStore is designed to distribute VMware Tools and agent updates seamlessly to multiple VMs. GuestStore enables the customers to distribute various types of content to the VMs, like an internal CDN system.
A shared datastore is used as a content repository where VI (Virtual Infrastructure) admins can place content like VMware Tools, various agents, binaries, and custom config files. Any type of shared datastore can be used as a content repository. Having content in a shared datastore gives centralized management capability. VI admins can also create a VM level rule set to regulate and monitor content distribution.
It is important to note that the VMware Tools GuestStore framework does not push the content to the VMs. GuestStore clients (Applications running in VMs) must pull the content as needed. Primarily, VMware Tools GuestStore is intended for distributing VMware specific content and any other content files smaller than 512MB in size.
With the VMware Tools GuestStore, it is now possible to release agent-specific updates, patches, and configuration files without a new VMware Tools release.
This was not possible earlier as an agent could not be released and updated without updating VMware Tools. Some agents also require access to some global configuration and customization for better integration in the customer environment. With VMware Tools Guest Store distributing all these agent components have become smooth and straight forward.
VMware Tools GuestStore in detail
With VMware Tools GuestStore, we have introduced some new components. Let us quickly look at them:
GuestStore Client: GuestStore clients are the applications, agents, or scripts running inside the VM that access content from GuestStore.
GuestStore Plugin: GuestStore plugin is shipped and installed with VMware Tools. GuestStore Plugin intercepts the connection request from the VM to the content repository (shared datastore). To avoid the VMX module from getting overloaded by VM requests, only one connection request is accepted at a time. All additional requests are queued.
VMX GuestStore Module: VMX GuestStore module proxies the request from the GuestStore plugin to GuestStore Daemon.
GuestStore Daemon: GuestStore daemon is a new service introduced in ESXi 7 U2. It is responsible for accepting download requests from the VMX GuestStore module, enforcing the content access policies, downloading the content from GuestStore, and sending it back to the GuestStore Client. GuestStore daemon is also responsible for caching the downloaded content to avoid duplicate download requests.
VMware Tools GuestStore workflow
In order to simplify the security model and also provide guest admin control over what gets into the guest, GuestStore only makes the content available to the guest without pushing anything automatically. Any agent that accesses GuestStore has to be installed in the guest manually or through some other mechanism outside GuestStore. Installing an agent (that accesses GuestStore) inside the guest by the guest admin acts as the opt-in by the guest admin for the agent and also for the GuestStore usage by the agent.
GuestStore access is enabled by default when GuestStore has been configured on the host. However, there are two VM settings that VI admins can use to control VM's access to the resources available in the GuestStore repository:
guestStore.access.deny=“semi-colon; separated;list;of;regular; expressions’to;deny; access”
When no rules are specified entire GuestStore content is accessible to the VM. When a VM requests content from GuestStore, grant rules are applied first followed by deny rules in the order specified. The rule that matches first gets applied and no further rules are considered. If there is no match for a resource, access is granted by default.
Example Access Rules:
- guestStore.access.grant=".*/vmtools-11/.*" and guestStore.access.deny=".*" → VM can access resources with ".*/vmtools-11/.*" pattern only and nothing else.
- guestStore.access.deny=".*/vmtools-11/.*" → VM cannot access resources with ".*/vmtools-11/.*" pattern, but it can access everything else.
- guestStore.access.grant=".*/vmtools-11/.*" and guestStore.access.deny=".*/vmtools-11/.*" → VM can access any resources with any pattern (deny is blocking what is granted).
- guestStore.access.grant=".*/vmtools-11/.*" and guestStore.access.deny=".*/vmtools-10/.*" is same as specifying just guestStore.access.deny=".*/vmtools-10/.*" → VM cannot access resources with ".*/vmtools-10/.*" pattern, but it can access everything else.
- If guestStore.access.grant=".*" is specified guestStore.access.deny is ignored completely because all resources will match grant rule.
All the activities related to VMware Tools GuestStore are logged under vmware.log of the guest and gstored.log of the host(/var/logs/gstored.log). This video demonstrates how the GuestStore Repository can be configured on a shared datastore.