vSphere tagging is all about adding metadata to a vSphere object. Doing so helps categorize similar vSphere objects by logically grouping them under a single label or tag.
Please refer this blog on tags and custom attributes for more information.
vSphere tags and custom attributes that were introduced in vSphere 4 has evolved tremendously over various vSphere releases.
The latest vSphere release, i.e., 7.0U1, has introduced significant performance improvements in the overall tags and custom attributes functionality. This latest white paper on vSphere tags and custom attributes covers all these improvements and recommendations in detail.
This blog will summarize all the recommendations for optimal performance of tags & custom attributes referenced in the above-linked whitepaper.
Creating a Tag
With vSphere 7.0U1, significant optimizations have been made to caching of tag information and database query mechanism. These optimizations have reduced the latency to create multiple tags and categories significantly.
VMware recommends creating multiple categories with fewer tags as compared to creating few categories with multiple tags. Alternatively, tag creation can be expedited by using multiple clients in parallel (multiple threads).
The following three API calls are primarily used/invoked to attach a tag to a VM(s).
Attaching a single tag to a single VM
Attaching a single tag to multiple VMs. VMware recommends using no more than 2000 VMs per call for best performance.
Attaching multiple tags to a single VM. For best performance, VMware recommends using no more than 2000 tags in a single call.
It is worth noting that for both attachTagToMultipleObjects() and attachMultipleTagsToObject() calls, as a best practice, VMware recommends not to use more than 2000 VMs and tags, respectively.
Exceeding this number will cause errors since the tagging service has limits regarding the number of items in a call. To validate if you are running into this issue, search the following error snippet in the log file for the vapi-endpoint service (i.e., /var/log/vmware/vapi/endpoint/endpoint.log):
“Received request (910,429 bytes) is bigger than the allowed request size (204,800 bytes) - request blocked!”
As an additional consideration, to avoid denial-of-service attacks to the tagging service, there is a limit of 1500 tagging-related operations per second, increased from 360 in vSphere 6.7 U3.
To validate if you are hitting this limit, search for error snippets like “Request rejected due to high request rate. Try again later.” in the vapi-endpoint log file (/var/log/vmware/vapi/endpoint/endpoint.log). In case your tag creation code exceeds this limit, VMware recommends inserting a pause of 2ms or more between subsequent tag creation calls.
Associating multiple tags to multiple VMs
Attaching multiple tags to multiple objects, attachMultipleTagsToObject() call should be used to improve performance.
The following table indicates that attachMultipleTagsToObject() is quite faster with a large number of VMs.
For large numbers of VMs and more than one tag, attachMultipleTagsToObject() scales the best. With one tag and many VMs, attachTagToMultipleObjects() scales the best.
Associating tag(s) to other vSphere inventory objects
Associating tags with other objects (hosts or datastores or content library items) follows the same ground rules as associating tags with VMs.
Let’s look at the following example scenarios to understand them better
- To associate 1 tag with 1 object (for example, host), VMware recommends using attach(tagID, HostID).
- To associate 1 tag with multiple hosts, VMware recommends using attachTagToMultipleObjects(tagID, HostID ).
- To associate multiple tags with 1 host, VMware recommends using attachMultipleTagsToObject(tagID, HostID).
- To associate multiple tags with multiple hosts, VMware recommends looping over hosts and calling attachMultipleTagsToObject(tagID, HostID) for each host.
vSphere tags and Enhanced Linked Mode (ELM)
In Enhanced Linked Mode (ELM), vCenter servers replicate tag and category information between peer nodes periodically. Each node replicates data with its peer every 30 seconds, so changes to categories and tag definitions on one node may take some time before propagating to the other nodes in the ELM setup.
For best performance, VMware recommends connecting ELM nodes in a star topology. Let’s take the following example ELM topology to understand the tags replication across various nodes.
In the above figure, the tags replication follows the following pattern:
• VC 1: VC 2 and VC 6
• VC 2: VC 1 and VC 3
• VC 3: VC 2 and VC 4
• VC 4: VC 3 and VC 5
• VC 5: VC 4 and VC 6
• VC 6: VC 1 and VC 5
When a new tag is added on VC 1, here is a rough timeline of replication:
- Time 0: tag added to VC 1
- Up to 30 seconds later, VC 1 replicates to VC 2 and VC 6
- Up to 30 seconds later, VC 2 replicates to VC 3; VC 6 replicates to VC 5
- Up to 30 seconds later, VC 3 replicates to VC 4; VC 5 also replicates to VC 4, but ELM performs conflict resolution to ensure only one new tag definition is created on VC 4.
As per the above timeline, in a 6-node ring topology, a tag may take up to 90 seconds to replicate throughout the system. Because each node pushes changes at 30second intervals, this is the worst-case latency. The nodes do not push changes simultaneously, so in the best case, a tag change may occur right before that node has been scheduled to push changes. In this case, the changes would be reflected more quickly.
vSphere tags can be of great help in managing and sorting large vSphere environments. Adhering to the vSphere tags best practices recommended by VMware ensures that customers get maximum performance.
For detailed information on vSphere tags, best practices, please this white paper.