Why share a GPU? The economics, the mechanics, the four mechanisms

Four mechanisms at a glance

The four mechanisms differ in where they divide the GPU — in time, in software, in hardware — and those differences determine what isolation guarantees each can offer.

• Time-slicing — The simplest form of sharing: the GPU driver serialises access across multiple CUDA contexts. No hardware changes required. High latency jitter. Suitable for batch and notebook workloads.
• MPS — Routes multiple CUDA client processes through a single server, enabling genuine concurrent kernel execution. Low-to-medium jitter. Suitable for trusted co-tenancy within a team. No fault isolation.
• MIG — Hardware partitioning on A100/H100/Blackwell. Up to seven fully isolated instances per GPU. The only mechanism with hardware-enforced memory, compute, and fault isolation.
• HAMi — Software fractional GPU via LD_PRELOAD API interception. Works on any CUDA-capable GPU. CNCF Sandbox project. Software isolation only — not equivalent to MIG.

Time-slicing

Time-slicing is the simplest form of sharing: the GPU driver serialises access across multiple CUDA contexts, switching between them on a configurable time quantum. Each context sees the GPU as if it owns it exclusively, but only for its time slice. No hardware changes are required; the mechanism is implemented in the NVIDIA kernel driver and exposed through the device plugin configuration in a Kubernetes environment.

CUDA Multi-Process Service (MPS)

MPS routes multiple CUDA client processes through a single server process. The MPS server submits work from all clients to the GPU concurrently, so kernel execution and memory-copy operations from different clients can genuinely overlap. On Volta-class hardware and later, each client gets a separate GPU address space, eliminating the address-space sharing risk of earlier MPS implementations. Clients share SM scheduling resources, meaning one misbehaving client can affect the throughput of others — there is no hard compute quota [7].

Multi-Instance GPU (MIG)

MIG is a hardware partitioning capability available on NVIDIA A100, H100, and Blackwell-class GPUs. It partitions a single physical GPU into up to seven fully isolated instances. Isolation is spatial and enforced in silicon: each instance’s streaming multiprocessors have separate and exclusive paths through the on-chip crossbar ports, L2 cache banks, memory controllers, and DRAM address buses [10][11]. A workload running in one MIG instance cannot observe the memory or execution of a workload in another, even if they share physical hardware.

HAMi fractional GPU

HAMi (Heterogeneous AI Computing Virtualization Middleware) is a CNCF Sandbox project, accepted in August 2024 [12]. It implements fractional GPU allocation through a software virtualisation layer: a shared library (libvgpu.so) is injected into container processes via LD_PRELOAD and intercepts CUDA driver and NVML API calls before they reach the hardware. The interception layer enforces per-container GPU memory and compute limits set by the scheduler, without requiring any hardware-level partitioning support.

The isolation matrix: four mechanisms × four axes

The four axes below are the operational properties that determine which mechanism fits which workload context. The table is a decision aid, not a ranking — there is no universally superior mechanism.

* MPS on Volta+ provides separate address spaces per client, but all clients share one MPS server process — a server crash terminates all clients.

• Time-slicing fault isolation is marked Partial because a GPU-reset event triggered by one context will typically clear all contexts on that GPU. Workloads in other time-slices are therefore exposed to each other's crash behaviour, but not to each other's in-flight data (contexts do not share address space).
• HAMi fault isolation is Partial for the same reason plus an additional caveat: if the intercept library itself faults, all workloads sharing that library instance may be affected.
• MIG's Negligible latency-jitter rating reflects the hardware-isolated L2 cache and memory controller paths. In practice, workloads in different MIG instances do not contend for memory bandwidth, which is the dominant source of jitter under concurrent inference load.
• HAMi's Low–Medium latency-jitter rating reflects the overhead of its LD_PRELOAD interception layer: every CUDA memory-allocation call passes through libvgpu.so before reaching the driver, adding a small but non-zero per-call cost. Unlike MIG, there is no hardware context switch; unlike time-slicing, execution is not serialised.

Hardware availability gates the choice

Matching mechanism to workload

Three workload archetypes dominate production AI platforms:

• Large distributed training runs — these typically benefit least from sharing. A well-utilised training job that fills a node’s GPUs is already efficient. Sharing here is counterproductive: it adds scheduling complexity without recovering significant idle capacity. Gang scheduling (ensuring all GPUs in a distributed job start together) matters more than partitioning. See the preceding article in this series on queue-aware scheduling for the relevant patterns.
• Online inference serving — typical VRAM utilisation for a serving replica depends heavily on model size and batch configuration. Many small-to-medium model serving deployments use 20–60 percent of a GPU’s VRAM, leaving substantial capacity unused. MPS (for trusted co-tenancy) or MIG (for isolated multi-tenancy) both apply here. Time-slicing is generally inappropriate because context-switch jitter degrades P99 latency.
• Notebook and experimentation workloads — these are the primary source of idle-GPU waste. A notebook that allocates a GPU on launch and then sits idle for hours while the user reads documentation is a textbook over-provisioning case. Time-slicing or HAMi fractional allocation constrains the impact of any one notebook on shared capacity. Fault isolation matters less here because notebook workloads are interactive and ephemeral.

What this series covers next

This article has introduced the mechanisms and framed the problem. The next article in this series (article 27, gpu-sharing-decision-tree) works through a structured decision tree that routes a specific workload and cluster context to the appropriate mechanism, including the trap branches — the configurations that look correct but fail under production conditions.