The GPU scheduling stack: queue admission, gang scheduling, and hardware abstraction in three layers

Layer 1 — Kueue: quota admission

Kueue is a Kubernetes-native job queueing system developed under SIG Scheduling, introduced to the Kubernetes ecosystem in October 2022. It does not replace the kube-scheduler — it sits in front of it, deciding whether a job is allowed to run at all before the scheduler ever sees the pods.

Four primitives

Kueue’s data model has four core primitives, documented in the Kueue concepts reference:

• ResourceFlavor — names a class of capacity (e.g. a node pool carrying A100 GPUs vs one carrying L40S GPUs). A ResourceFlavor maps to node labels and tolerations.
• ClusterQueue — the quota gate at cluster scope. A ClusterQueue owns a slice of the ResourceFlavor's capacity: e.g. team-a may use up to 8 nvidia.com/gpu units.
• LocalQueue — a namespace-scoped pointer into a ClusterQueue. Teams submit jobs to their namespace's LocalQueue; the LocalQueue forwards admission requests to the ClusterQueue.
• Workload — the admission object wrapping a Job, JobSet, PyTorchJob, or similar. Kueue holds the Workload in a Pending state (with the underlying pods suspended) until the ClusterQueue has capacity. When capacity is available, Kueue unsuspends the Workload and the scheduler sees the pods for the first time.

What Kueue does not do

Kueuedoes not place pods. It holds or releases a job; once released, placement is entirely the scheduler’s concern. This is stated explicitly in the ClusterQueue concepts documentation. The implication: if you need all pods of a distributed job to land simultaneously (gang scheduling), you need an additional mechanism at the placement layer — Kueue’s release of a Workload does not guarantee atomic placement.

Layer 2 — Volcano: gang scheduling and placement

Volcano is a CNCF incubating batch scheduling project (CNCF projects page). It extends Kubernetes with a batch scheduling framework that adds two capabilities the default kube-scheduler does not provide: gang scheduling and topology-aware placement.

Why gang scheduling matters for distributed training

A distributed training job running PyTorch DDP or DeepSpeed requires all worker processes to start before any of them can begin the first communication round. Without gang scheduling, a partial allocation is possible: some workers start, others cannot schedule because the remaining nodes are occupied, and the running workers hold their GPU allocations while waiting for peers that never arrive. This is a starvation deadlock — documented in the Volcano project documentation. Volcano’s response is the PodGroup: all members of the group must be satisfiable before any pod is bound to a node.

Key primitives

• PodGroup — the atomic scheduling unit. The minMember field sets how many pods must be satisfiable before any are bound. Setting minMember lower than the job's actual worker count defeats the gang guarantee.
• Queue — a Volcano-level fair-share queue with configurable weights. Multiple teams can share a cluster-level capacity pool; the Queue CRD carries a capability ceiling and a reclaimable flag (allowing other teams to borrow idle quota).
• TopologyPolicy — routes workers to nodes that share NVLink or InfiniBand, reducing inter-node communication latency on multi-node training. The preferSingleSocket value is the practical default for most GPU node configurations. The preferSingleNUMANode value may cause jobs to wait indefinitely on nodes where GPUs span multiple NUMA domains (a common topology on multi-socket servers with more than four GPUs); verify your node NUMA topology with numactl --hardware before using this value in production.

How Kueue and Volcano compose

The two systems compose without conflict. Kueue manages quota — whether a job is allowed to consume resources at all. Volcano manages placement — whether all the pods of an admitted job can land simultaneously. The handoff point is the Workload unsuspend event: when Kueue determines quota is available and lifts the suspension, the pods become visible to the scheduler; if the scheduler is Volcano, it then applies PodGroup gang semantics before binding any pod.

Layer 3 — the GPU Operator: hardware abstraction

The NVIDIA GPU Operator is a Kubernetes operator that bundles the full software stack a GPU node needs: the driver, NVIDIA Container Toolkit, device plugin, Node Feature Discovery, GPU Feature Discovery, DCGM exporter for metrics, and MIG manager. One Helm release replaces a stack of host-level installs and independent DaemonSets.

What the device plugin does

The NVIDIA device plugin registers nvidia.com/gpu as an extended resource with the kubelet on each node. Once registered, pods can request it like any other resource in a container spec. The count exposed depends entirely on how the operator is configured: a node with 8 physical A100s might advertise 8 units (whole-GPU mode), 64 units (time-slicingwith 8× replication per card), or a set of MIG slice units (e.g. 56 units under the 1g.10gb profile across 8 GPUs).

DCGM exporter and observability

The bundled DCGM exporter emits GPU metrics directly into a Prometheus scrape endpoint. The metrics most useful for platform utilisation tracking are DCGM_FI_DEV_GPU_UTIL (SM utilisation percentage), DCGM_FI_DEV_FB_USED (framebuffer memory used), and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (tensor core active percentage — the most direct signal for whether a training job is doing useful compute). Without DCGM exporter, GPU utilisation is invisible to the platform; the first sign of a problem is usually a job that runs for far longer than expected.

Driver installation strategy

The operator’s driver component is correct for bare-metal nodes where no driver is pre-installed. On managed Kubernetes distributions that ship a pre-installed GPU driver (a common pattern on managed cloud node pools), driver.enabled=false is the safe configuration — it allows the operator to provide the device plugin, DCGM, and MIG manager without conflicting with the already-loaded driver module. Mixing operator-managed and pre-installed drivers on the same node leaves the node in a NotReady state due to module-load conflicts.

A job flowing through all three layers

Following a PyTorchJob submission from a data science team makes the layer interactions concrete:

1. 1The team submits a PyTorchJob to their namespace. The Kueue webhook intercepts the creation and immediately suspends all pods. Kueue wraps the job in a Workloadobject and queues it against the team’s LocalQueue.
2. 2Kueue checks the team’s ClusterQueue. If the requested nvidia.com/gpu count fits within the remaining quota (including any borrowable capacity from a cohort), Kueue unsuspends the Workload.
3. 3The unsuspended pods are visible to the scheduler. If the cluster uses Volcano as its scheduler (via schedulerName: volcano in the pod spec), Volcano checks the PodGroupassociated with the job. The PodGroup’s minMember value must be satisfiable — all workers placeable at the same time — before any pod is bound.
4. 4Once all pods are placeable, Volcano binds them atomically to GPU nodes. The kubelet on each target node consults the device plugin’s resource registry to allocate the exact GPU units (whole cards, MIG slices, or time-sliced replicas) to the container.
5. 5The NVIDIA Container Toolkit injects the allocated GPU devices into the container’s cgroup. The training process starts with exclusive access to the allocated GPU units.

A stack overview (diagram)

The responsibility boundary at each layer:

Three common failure modes: which layer fails first

Understanding the layer boundaries makes it faster to diagnose production problems. Each failure mode has a clear first-failure layer:

Configuration sketch

The following fragments illustrate the three-layer wiring. They are illustrative — tune namespaces, resource counts, and GPU SKU labels to your environment.

# Layer 1: Kueue — quota admission
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-a-queue
spec:
  namespaceSelector: {}
  cohort: shared-gpu-pool
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: gpu-a100
          resources:
            - name: nvidia.com/gpu
              nominalQuota: "8"       # guaranteed share
              borrowingLimit: "4"     # may borrow up to 4 extra
  preemption:
    reclaimWithinCohort: Any          # reclaim guaranteed share if borrowed

# Layer 2: Volcano — gang scheduling
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: pytorchjob-workers
  namespace: team-a
spec:
  minMember: 4            # ALL 4 workers must be placeable before any are bound
  minResources:
    nvidia.com/gpu: "4"
  queue: team-a-volcano-queue
  priorityClassName: gpu-training

# Layer 3: GPU Operator — hardware abstraction
# Bare-metal / self-managed node example (driver.enabled=true)
driver:
  enabled: true           # set false on nodes with a pre-installed driver
toolkit:
  enabled: true
devicePlugin:
  enabled: true
dcgmExporter:
  enabled: true
  serviceMonitor:
    enabled: true         # scrape by Prometheus Operator
migManager:
  enabled: true           # A100/H100 MIG partitioning
nodeStatusExporter:
  enabled: true
# Resource flavor label used by Kueue ResourceFlavor nodeLabels
nodeFeatureDiscovery:
  worker:
    tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule

Quota borrowing and preemption

Kueue’s cohort model allows multiple ClusterQueues to share a borrowing pool. A team with a nominalQuota of 8 GPUs can borrow additional capacity from another team’s idle allocation, up to a configured borrowingLimit. When the lending team submits a new job and needs its quota back, preemptionreclaims it. This is the mechanism that makes a fixed GPU pool feel larger than its physical count — idle quota does not sit unused while other teams’ jobs wait.

The Kueue metrics to monitor for this behaviour are kueue_pending_workloads (rising count indicates queue pressure), kueue_admitted_workloads_total (rate of successful admission), and kueue_quota_reserved_resources (current utilisation against nominalQuota per ClusterQueue).

Alternatives to this stack composition

The three-layer stack (Kueue + Volcano + GPU Operator) is one composition. Others are viable depending on workload mix:

• Kueue + default scheduler + GPU Operator — appropriate for single-pod or small-scale jobs that do not require gang semantics. Lower operational overhead; loses topology-aware placement and gang guarantees.
• Kueue coscheduling plugin + GPU Operator — the Kubernetes SIG Scheduling coscheduling plugin provides gang admission as a scheduler plugin rather than a separate scheduler. Less operationally heavy than Volcano; less mature for topology-aware placement.
• Apache YuniKorn + GPU Operator — an alternative batch scheduler with gang scheduling and capacity management. Broader workload support (Spark, Flink natively), less adoption in the Kubernetes MLOps ecosystem as of 2026.
• Standalone device plugin DaemonSet (no GPU Operator) — possible but requires managing the driver, container toolkit, and DCGM exporter separately. Increases operational surface at the hardware abstraction layer.

What comes next: the GPU-sharing decision tree

The stack described above treats each nvidia.com/gpu unit as a whole-card allocation by default. Whether to partition those cards — and which mechanism to use (time-slicing, MPS, MIG, or fractional GPU virtualization) — is a separate decision that belongs at the GPU Operator layer. That decision has a significant effect on the isolation and latency jitter profile seen by jobs. The next article in this series, the GPU-sharing decision tree (Article 27), provides a structured framework for that choice — starting from workload type (training vs inference) and hardware generation (MIG-capable vs not).