Bitfusion FAQ

​AI/ML applications are expanding exponentially in every vertical: finance, medical, automotive, security, retail, manufacturing, and on and on. Machine learning is used for image recognition, natural language processing, identifying patterns in tabular data, and any dataset, really.

​AI/ML requires a tremendous amount of computation, but it is easy to perform in parallel. Hardware acceleration, principally GPUs today, is practically a requirement to perform the computation quickly enough in acceptably low power envelopes.

​First, is cost: GPUs are both expensive and underutilized. Some studies show they are used only 15% of the time.

​Second is the bare metal barrier": AI/ML started in bare metal silos. To scale AI/ML needs the flexibility of virtualization and the orchestration of modern applications.

​Virtualization of GPUs increases utilization and brings flexibility, agility, monitoring, management, and many other benefits. Plus, Bitfusion runs as an application and thus, automatically runs in containerized environments.

​Bitfusion is GPU virtualization solution for AI/ML workloads. It is a feature of vSphere. It pools the GPU servers and allows clients to allocate whole or partitioned GPUs from the pool across the network.

​A Bitfusion server is an appliance (a VM under vSphere made from an OVA) that runs the Bitfusion server software. Each Bitfusion server requires physical GPUs passed through from the host (with DirectPathIO). All Bitfusion servers must be pooled under a single vCenter instance.

A Bitfusion client is any VM, bare-metal machine, container, or K8S pod that installs the Bitfusion client software – a DEB or RPM package – and that has been issued a token to authorize its requests to the Bitfusion servers.

​Bitfusion is intended for AI/ML workloads. Bitfusion only runs CUDA applications.

​Bitfusion works by running an existing application and intercepting its calls to CUDA, NVIDIA’s computational API. Intercepted calls are sent to remote resources (GPUs) on a Bitfusion server that were allocated when the application was launched.

​Bitfusion is NOT for use with graphical applications – the non-CUDA, graphical API calls are not intercepted.

Because of the intermittent and unpredictable variation in network performance/latency, Bitfusion is not suitable for real-time applications (including VDI). If your application has a relatively large “real-time” envelope (say, human-level, conscious reaction time) then Bitfusion can still be considered.

​No, applications run as is, mostly on the client machine, allocating and accessing memory and storage as before. The only difference is their CUDA calls are virtualized.

No, storage is only accessed directly by the clients. The virtualization of the CUDA calls will transfer necessary data to the devices.

​Bitfusion runs on Linux only, and for the x86 instruction set. MS Windows is not on the roadmap. Recent distributions of Ubuntu, RHEL, and CentOS are tested and supported. See the VMware Interoperability Matrix for details.

​You will need 10 Gbps networking or higher (not a functional requirement, but strongly advised for performance). Consider higher bandwidth especially for clients allocating more than two GPUs.

​You should target a client-server latency of 50 microseconds or less (again, for performance).

​TCP/IP, RoCE, Infiniband (IB). Note: this includes the VMXNET3 and PVRDMA virtual adapters, along with NICs or HBAs you may pass through.

Technically, Bitfusion is made to be compatible with certain versions of the NVIDIA driver, not the GPUs directly. But the GPUs we test against are also listed in the VMware Interoperability Matrix.

Yes, even within a single server. We encourage variety within a cluster. However, we recommend caution when installing different GPU types and sizes on a single server. Applications using multiple GPUs may not be written to deal with heterogenous types, nor with the differences in their capabilities. Bitfusion will only allocate the memory of the smallest GPU. These intra-server quirks could be avoided by filtering (see FAQ question on filtering), but will not arise in the first place if all the GPUs on a single server are the same.

There are no hard-coded limits to the number of clients nor the number of servers in a Bitfusion cluster. However, we do not expect that the number of servers will scale forever without performance implications. We advise you to perform empirical testing if you wish to scale beyond 20 or 25 servers. We do not see such limits on the number of clients.

No. At this time, all Bitfusion servers under a vCenter instance form a single Bitfusion pool.  The Bitfusion servers may be instantiated on hosts in different vSphere clusters, but they will act as a single cluster from a Bitfusion point of view.

​Clients issue a separate Bitfusion command for each application they run (or for a session of runs). The command has a mandatory switch specifying the number of physical GPUs requested and an optional switch for the partition size from each physical GPU.
Clients may have multiple, concurrent runs and sessions.

​Any request for some number of GPUs must be satisfied by separate physical GPUs from a single Bitfusion server. However, since a client can run concurrent Bitfusion processes, the aggregate GPU count from a client could run as high as the total GPU count in the pool (though it would be an unusual use case).

​Bitfusion partitions and isolates the GPU framebuffer memory for each client request. Scheduling of the computation resources is left to the CUDA hardware and software.

No. For any single request, each partition must come from a separate physical GPU (and all on one server).

​Yes. For example, you can request resources after filtering for server address ranges, for GPU type, for RDMA connection, frame buffer size and more.
See the Bitfusion User Guide for more information.

​You can set GPU maximums, for all and for individual clients. This scores partials too—if a client has a limit of 2.5 GPUs, for example, it could run three simultaneous jobs allocating, respectively, 1 GPU, 0.75 of a GPU and 0.75 of a GPU, before reaching the limit and before seeing a fourth request fail.

​You may set an idle time limit on a client-by-client basis in the Bitfusion management window (of vCenter). GPUs that sit idle longer than the limit will be deallocated.

​No, not for a given application. But applications do not expect this of physical devices either.

Yes.

​The underlying principle is that Bitfusion software is just another application – and containers were made to contain applications. Since Bitfusion does not have kernel or ESXi components, there is nothing to integrate – just containerize it. Kubernetes and TKG just sit at the layer above, orchestrating multiple containers.

​Further, we do a large amount of Bitfusion quality assurance within containers.

​Further still, we support the authorization of Bitfusion client pods via the Kubernetes secrets facility

​Bitfusion does not have a formal scheduler. All servers respond to a request and the client will select among the responses that can satisfy the request. If no server has sufficient, currently-free resources, the client will spin reissuing the request with a limited, exponential back-off.

​No, a time limit can be specified.

​We recommend using a third-party scheduler such as Slurm or LSF. As Bitfusion is just another application, integration is simple.

Yes. Bitfusion Scheduling – A Day at the Races

​It depends (when doesn’t it?). Each application is different. Large batch sizes, large models are conducive to good performance (low overhead from Bitfusion). Bitfusion uses pipelining, command re-ordering, and other techniques, to keep the GPU computation resources busy while data is transferred in the background; this can hide much of the network latency. Applications with a high ratio of compute-to-communication perform the best.

The ones with the largest effect are typically:

• Use larger batch sizes
• Use a large MTU
• Configure for the lowest network latency possible
• Use an RDMA transport

​The command fails.

​The application will perceive the disappearance of the device from the bus, typically ending the application. Many applications allow you to take checkpoints, so you may restart it with Bitfusion allocating a GPU from a different server.

​The remaining Bitfusion servers will continue to function as long as a majority of the original servers are healthy.

​That’s not an “error” question, but okay, why make a whole separate section for this.
Live migration will work on Bitfusion clients.

​Live migration is not allowed on Bitfusion servers (its GPUs use DirectPathIO, which prevent it).

​All you need is a vSphere 7 Plus evaluation license. Get the downloads and documentation at our landing page.
https://docs.vmware.com/en/VMware-vSphere-Bitfusion/index.html

​Yes.
https://core.vmware.com/bitfusion

​All future looking statements represent no commitment from VMware to deliver features or products. Everything is subject to change whether due to technical feasibility, demand, or other factors.

​VMware looks to bring greater generality to the virtualization of acceleration technologies—not just hardware, but software too.  This includes support for multiple vendors.

​For one approach, read about Project Radium: https://core.vmware.com/blog/application-remoting-energize-radium-not-dilithium.