HAMi — fractional GPU on the GPUs you actually have

The isolation contract: what is guaranteed and what is not

The practical value of HAMi— and the reason it requires careful placement decisions — depends entirely on understanding where the isolation boundary is hard and where it is approximate.

Resource names and a worked pod spec

HAMi’s device plugin registers four resource names per GPU node. The full set is documented in the HAMi configuration reference: [5]

• nvidia.com/gpu — integer virtual GPU slot count (how many fractional slices the pod needs)
• nvidia.com/gpumem — device memory hard-cap per slot, in MiB
• nvidia.com/gpucores — compute soft-cap per slot, in percent (0–100)
• nvidia.com/gpumem-percentage — alternative to gpumem, expressed as a fraction of total card memory

A workload running a 7B-parameter int4 LLM that needs approximately 4 GiB on a 16 GiB card:

resources:
  limits:
    nvidia.com/gpu: "1"          # one virtual GPU slot
    nvidia.com/gpumem: "4096"    # 4 GiB hard memory cap
    nvidia.com/gpucores: "25"    # 25 % SM compute soft-limit

With those settings, four such pods can share a single 16 GiB card: each gets 4 GiB with a hard ceiling, and each is nominally throttled to 25% of the streaming multiprocessors. A fifth pod requesting 4096 MiB will not be scheduled onto that card — the scheduler extender’s Filter phase removes it from consideration because the remaining unallocated memory falls below the request.

Deploying HAMi via Helm

HAMi ships a Helm chart. The steps below apply to any CNCF-conformant Kubernetes distribution; the only distribution-specific element is how the scheduler extender configuration reaches the kube-scheduler binary. The official deployment guide covers additional options.

kubectl label nodes <gpu-node-1> gpu=on
kubectl label nodes <gpu-node-2> gpu=on

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
extenders:
  - urlPrefix: "https://127.0.0.1:443"
    filterVerb: "filter"
    bindVerb: "bind"
    weight: 1
    nodeCacheCapable: true
    managedResources:
      - name: "nvidia.com/gpu"
        ignoredByScheduler: true

helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update

helm upgrade --install hami hami-charts/hami \
  --namespace kube-system \
  --set scheduler.extender.urlPrefix="https://127.0.0.1:443" \
  --set scheduler.nodepolicies="binpack" \
  --set scheduler.gpupolicies="binpack" \
  --version "2.7.0"

Stacking quota-aware queuing on top of HAMi

HAMi handles within-node fractional allocation. It does not, by itself, prevent a single team from submitting enough pods to exhaust the GPU pool for the rest of the cluster. Kueue’s official guide for using HAMi with Kueue describes the integration: HAMi’s resource names are first-class Kueue resources, and a ResourceTransformation object instructs Kueue to compute aggregate quota consumption by multiplying per-slot values by the vGPU count. [6]

For example: a pod requesting nvidia.com/gpu: 2 and nvidia.com/gpumem: 1024 contributes 2048 MiB to the team’s total-gpumem quota counter. This prevents a team from issuing many small-memory pods that individually fit per-pod quota limits but collectively exhaust the physical pool.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-inference
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources:
        - nvidia.com/gpu
        - nvidia.com/gpumem
        - nvidia.com/gpucores
      flavors:
        - name: hami-gpu
          resources:
            - name: nvidia.com/gpu
              nominalQuota: "8"      # 8 virtual GPU slots
            - name: nvidia.com/gpumem
              nominalQuota: "32768"  # 32 GiB total across the team
            - name: nvidia.com/gpucores
              nominalQuota: "200"    # 200 % SM (2 full cards nominal)

The admission order matters: the quota-aware admission webhook runs before the HAMi mutating webhookin the Kubernetes admission chain. The queue evaluates whether the team has quota; if admitted, HAMi’s webhook then rewrites the scheduler name and injects the libvgpu.sovolume mount. If the queue suspends the workload (quota exhausted), the pod never reaches HAMi. Align the ClusterQueue’s nominalQuota for gpumem with the actual sum of physical card memory on your GPU nodes — setting it higher creates a queue that admits workloads that the extender will then fail to place.

Observability

HAMi’s Helm chart does not bundle a GPU hardware exporter. Wire the NVIDIA DCGM exporterseparately — it reads hardware performance counters from the GPU’s PMU and is unaware of HAMi’s virtual allocation boundaries, so it surfaces physical ground truth regardless of whether HAMi is managing the allocation. HAMi-specific per-allocation telemetry is exposed from the hami-scheduler pod’s /metrics endpoint on port 8080. HAMi’s dashboard documentation lists Grafana dashboard IDs 22043 and 21833 for visualising this data. [7]

The two-layer scrape strategy gives the full picture:

• DCGM exporter — DCGM_FI_DEV_GPU_UTIL (physical compute utilisation), DCGM_FI_DEV_FB_USED / FB_FREE (physical frame-buffer consumption), DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (tensor core activity, the primary training efficiency signal).
• HAMi scheduler metrics endpoint — per-container allocated memory and core-utilisation accounting. Comparing the HAMi accounting view against the DCGM physical view surfaces cases where a container's actual consumption diverges from its declared request.

Alert on DCGM_FI_DEV_FB_USED approaching the sum of all active containers’ gpumemlimits — this signals that the HAMi accounting and the hardware reality are converging and a new pod may fail placement even if the extender believes capacity is available.

Operational diagnostics

# Inspect extender filter decisions
kubectl logs -n kube-system -l app=hami-scheduler --tail=100

# Check scheduler events on the stuck pod
kubectl describe pod <pending-pod> | grep -A 10 "Events:"

# CUDA_DEVICE_MEMORY_LIMIT must be present in the container environment
kubectl exec -it <pod> -- env | grep CUDA_DEVICE_MEMORY_LIMIT

# If absent: check that the hami-webhook pod is healthy
kubectl get pods -n kube-system -l app=hami-webhook

# Verify the webhook's namespaceSelector includes the pod's namespace
kubectl get mutatingwebhookconfigurations hami-webhook -o yaml | grep -A5 namespaceSelector

Comparison with time-slicing, MPS, and MIG

HAMi occupies a specific position in the GPU-sharing mechanism space. The table below summarises the four mechanisms available on NVIDIA hardware:

HAMi is the appropriate mechanism when: the node carries a GPU generation that does not support MIG (T4, L4, A10G, V100); the workload mix is inference-dominated with known, stable memory footprints; and memory isolation between tenants is required but hardware-level fault isolation is acceptable to trade away.

When MIG is available, prefer MIG

On A100 and H100 nodes, hardware MIGpartitioning provides memory isolation, compute isolation, and fault isolation that are each enforced at the silicon level — none of which can be achieved through userspace library interception. If your cluster carries MIG-capable GPUs and the administrative cost of managing MIG profiles is acceptable, MIG is the stronger mechanism for multi-tenant inference.

HAMi’s value proposition is specifically on the non-MIG GPU generations that constitute the majority of most existing clusters. It bridges the isolation gap between the unprotected sharing of time-slicing and the hardware partitioning of MIG— for the GPU inventory that cannot use MIG at all.

HAMi’s CNCF Sandbox status (accepted August 2024, with an Incubation application filed July 2025) reflects an active but maturing community. Evaluate its graduation trajectory against your own production standards before committing a critical-path inference fleet to it.