Deep dive into VirtualMachineService
With vSphere 7 Update 2a release, we released VM Service that enables a Kubernetes-native way of provisioning and managing virtual machines. This feature enabled developers to run modern applications on vSphere with Tanzu. There are various aspects of the feature covered in VMware blogs such as this. This blog is intended to provide a deeper understanding of one of the most important aspects of this feature called VirtualMachineService. A note on disambiguation, “VM Service” is the feature name and is different from “VirtualMachineService”, the topic of this post.
One of the primary goals of VM Service is to allow VM-based workloads to be able to expose their services to other workloads within, or external to, the Supervisor Kubernetes cluster. For example, a MySQL instance running in a VM form factor should be able to provide database services to all workloads, regardless of the client workloads running in Kubernetes pods or another VM. Kubernetes solves this problem for pods using a core concept called Service that exposes a stable, load-balanced, network endpoint that is backed by service that is provided by pod(s). The Service abstraction allows workloads within and external to the Kubernetes cluster to consume the service provided by pod(s). VM Service solves this problem for VMs by providing an abstraction called VirtualMachineService.
A VirtualMachineService allows a VM to provide a service to other workloads using a stable, virtual IP address in Supervisor cluster. Using label selector, a set of VMs can collectively provide load-balanced services to other workloads. VirtualMachineService is a custom resource (CRD) on Supervisor cluster and vm-operator, a component of VM Service, contains a controller that actuates this and provides integration with other Supervisor components. Like Kubernetes Service, there are two flavors here: type LoadBalancer and type Cluster IP. However, the current version of the product supports only VirtualMachineService of type LoadBalancer which uses the underlying Load Balancer solution integrated with the Supervisor cluster (vSphere networking stack with HAProxy/AVI or VMware NSX-T Data Center).
Let’s see this in action. In my lab, I have a namespace “demo-ns” and I am creating a VirtualMachineService object. Note that I have specified a label selector called “app: my-app" so that every matching VM will be a member of this LoadBalancer. I have also specified 80 to be the fronting port of the LoadBalancer and the port on which the workload (VM) is listening.
root@421ea609aee516856a6179fd255c8a42 [ ~ ]# cat svc.yaml
apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachineService
metadata:
name: my-lb
namespace: demo-ns
spec:
selector:
app: my-app
type: LoadBalancer
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
On applying this YAML, we see that in addition to the VirtualMachineService object, we see Service and Endpoint objects with the same name. This is because VM Service controllers created a headless Kubernetes Service to associate with VirtualMachineService and this contains the external IP address. The Endpoints aren’t populated yet because there are no VMs that match the label selector.
root@421ea609aee516856a6179fd255c8a42 [ ~ ]# kubectl get virtualmachineservice -n demo-ns
NAME TYPE AGE
my-lb LoadBalancer 3d19h
root@421ea609aee516856a6179fd255c8a42 [ ~ ]#
root@421ea609aee516856a6179fd255c8a42 [ ~ ]# kubectl get service -n demo-ns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-lb LoadBalancer 172.24.129.140 192.168.0.6 80:31944/TCP 3d19h
root@421ea609aee516856a6179fd255c8a42 [ ~ ]#
root@421ea609aee516856a6179fd255c8a42 [ ~ ]# kubectl get endpoints -n demo-ns
NAME ENDPOINTS AGE
my-lb <none> 3d19h
root@421ea609aee516856a6179fd255c8a42 [ ~ ]#
Since a headless Service isn’t very helpful, let’s go ahead and create some VMs in “demo-ns” namespace. I have already associated a few VM classes and a Content Library containing a VMware-validated Ubuntu image to “demo-ns”. I am creating two VirtualMachine objects “vm-1” and “vm-2” specifying a label called “app=my-app" for both. Here is the YAML I am using for both the VMs. For brevity, I am omitting the contents of the bootstrap ConfigMap “my-config” that contains the CloudInit standard cloud-config blob containing the username and password for the guest OS.
root@421ea609aee516856a6179fd255c8a42 [ ~ ]# cat vm.yaml
apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachine
metadata:
name: vm-1
namespace: demo-ns
labels:
app: my-app
spec:
networkInterfaces:
- networkName: "primary"
networkType: vsphere-distributed
className: best-effort-small
imageName: ubuntu-20.04-vmservice-v1alpha1.20210528
powerState: poweredOn
storageClass: gc-storage-profile
volumes:
- name: myRootDisk
vSphereVolume:
deviceKey: 2000
capacity:
ephemeral-storage: "10Gi"
vmMetadata:
configMapName: my-config
transport: OvfEnv
On applying this YAML, the VMs gets created and powered-on. You can see that they acquired an IP address on the specified network.
root@421ea609aee516856a6179fd255c8a42 [ ~ ]# kubectl get vm -n demo-ns -o wide
NAME POWERSTATE CLASS IMAGE PRIMARY-IP AGE
vm-1 poweredOn best-effort-small ubuntu-20.04-vmservice-v1alpha1.20210528 192.168.128.10 4m29s
vm-2 poweredOn best-effort-small ubuntu-20.04-vmservice-v1alpha1.20210528 192.168.128.11 4m19s
root@421ea609aee516856a6179fd255c8a42 [ ~ ]#
If we go back and examine the Endpoint object that was associated with the VirtualMachineService, we see that there are two new addresses added to it. These are the IP addresses from the matching VMs that got added to the Endpoint’s subsets. As VirtualMachine resource keeps getting created and deleted, VirtualMachineService controller automatically updates the backing Endpoints without any user intervention.
root@421ea609aee516856a6179fd255c8a42 [ ~ ]# kubectl get endpoints my-lb -n demo-ns
NAME ENDPOINTS AGE
my-lb 192.168.128.10:80,192.168.128.11:80 3d20h
root@421ea609aee516856a6179fd255c8a42 [ ~ ]# kubectl get endpoints my-lb -n demo-ns -o jsonpath='{.subsets[0].addresses}' | jq
[
{
"ip": "192.168.128.10",
"targetRef": {
"apiVersion": "vmoperator.vmware.com/v1alpha1",
"kind": "VirtualMachine",
"name": "vm-1",
"namespace": "demo-ns",
"uid": "c2685b24-efd1-4bdf-9b5b-2a4ee18b8904"
}
},
{
"ip": "192.168.128.11",
"targetRef": {
"apiVersion": "vmoperator.vmware.com/v1alpha1",
"kind": "VirtualMachine",
"name": "vm-2",
"namespace": "demo-ns",
"uid": "b17cb9cc-d97a-4969-bd50-1117de98a1e8"
}
}
]
root@421ea609aee516856a6179fd255c8a42 [ ~ ]#
If you were to ping the LoadBalancer IP address 192.168.0.6, you will notice that we hit vm-1 and vm-2 alternatively. That’s the Kubernetes Service routing the traffic in a round-robin fashion. Scaling-out your application is as simple as adding more VMs with a matching label selector and scaling-in is the inverse. There is no need to configure any Kubernetes objects other than VirtualMachine. This allows developers to easily manage their VM-based applications on Supervisor cluster.
VM Service also manages the nodes for Tanzu Kubernetes clusters and internally leverage VirtualMachineSevice for providing load balancer virtual IP address to the control plane. In addition to this, VirtualMachineService is used internally to actuate the LoadBalancer-type and ClusterIP-type Service objects created inside Tanzu Kubernetes clusters.
In conclusion, VirtualMachineService provides a simple and powerful mechanism to manage the set of IP addresses representing the VMs that back a workload. It automatically ensures that only existent and healthy endpoints back it and achieves a load-balanced virtual IP in a distributed application.