github.com/Zhanyl-tech/k8s-gpu-scheduler-lab · Python · MIT

partial Built: the kind + kwok substrate, the workload generator, the metrics, K0 (the default kube-scheduler) and four degenerate baselines, and the Phase 2 measurement harness. No cluster run has been made with the Phase 2 harness, so every cluster number is Phase 1's, from single runs of K0, D-fifo, D-random and D-largest; D-preempt, added in Phase 2, has run only in the in-process model. Not built: Kueue, Volcano, NVIDIA's Volcano bin-packing and KAI with DRA, which are the comparison itself.

Slurm and Kubernetes solve the same problem, allocating scarce accelerators across competing jobs, with different machinery. Slurm uses one controller: multifactor priority plus backfill. Kubernetes splits it across projects: Kueue does quota and admission, Volcano gang scheduling and DRF fairness, NVIDIA’s KAI topology-aware gang scheduling with DRA. Each is usually benchmarked, when at all, against the default scheduler on a workload of its own choosing.

This lab runs them on the same trace and the same fleet. I have not found a published controlled comparison of the Kubernetes options on an identical trace and substrate. The closest I found is Radiant’s comparison of Kueue, Volcano and Slinky (2026-04-23), which used the same mock jobs but different hardware for Kueue (2× H200) than for the other two (2× H100). What this lab adds is the control: one trace, one fleet, stated metric definitions, and degenerate baselines in every table.

What kwok can and cannot tell you

Every GPU node is a kwok node: an API object with no kubelet behind it, advertising nvidia.com/gpu it does not have. The control plane is real, the scheduler is real, and every binding decision is real. Nothing else is. So the lab can measure placement, queue wait, admission order, packing, fragmentation and gang behaviour, and it cannot measure NCCL performance, real GPU contention, MIG/MPS sharing, thermal behaviour, or whether a topology-aware placement was actually faster. A scheduler that wins here has made better decisions on this trace; it has not been shown to make jobs faster.

The claim it was started to test

NVIDIA reported that adding bin-packing to the Volcano scheduler reduced GPU fragmentation on one production cluster. The primary source is NVIDIA’s technical blog, Practical Tips for Preventing GPU Fragmentation for Volcano Scheduler (Ameya Parab, 2025-03-31). On a DGX Cloud Kubernetes cluster of four-GPU L40S nodes, compared with Volcano’s default gang placement, it reports nodes with all four GPUs free going from 18 to 214 and average GPU utilisation of roughly 90%. It gives no node count, no workload, and no percentage reduction in fragmentation.

The figure this repo was started with, “a 34% reduction versus the default scheduler, shared at KubeCon EU 2026”, is unverified in both value and attribution. The only place I found it is a third-party blog post (Gheware, 2026-05-13), which credits “benchmarks shared at KubeCon EU 2026” and names no talk, speaker, slides or recording. NVIDIA’s post does not mention KubeCon or give any percentage, and the lab has not tested the figure. (Both pages re-read 2026-09-27.) The quantity NVIDIA did report, how nodes are distributed by free-GPU count, is computable from the samples the lab records; the metric is not implemented yet.

Its most useful finding so far is about a metric

Fragmentation is reported under three definitions because they disagree. Under the queue-relative definition (A), the largest-first baseline read 0.0%, which sounds like perfect packing; under a fixed reference request (B) the same run read 55.6% (Phase 1, kind + kwok control plane, one run). Largest-first starves small jobs, so a one-GPU pod is always pending and no free GPU ever counts as unusable under A. A fragmentation number that does not state its definition cannot be reproduced or disputed, and that includes the 34% above. A third, structural definition (C) is shared with slurm-scheduler-lab through a common golden test vector, so the Slurm and Kubernetes sides compute it identically.

Configurations

idschedulerstatus
K0default kube-scheduler + device pluginbuilt
D-fifo, D-random, D-largest, D-preemptdegenerate baselines: a scheduler that does not clearly beat D-random has not been shown to schedulebuilt
K1+ Kueue (quota, admission)planned
K2+ Volcano (gang, DRF)planned
K3+ NVIDIA’s Volcano bin-packingplanned
K4NVIDIA KAI + DRAplanned
S0Slurm on the same trace, via slurm-scheduler-labpartial (model side built there)

Phase 2: fixing how numbers are produced

Phase 1’s results showed the harness, not the schedulers, was the largest source of variation: the same K0 configuration on the same trace and seed gave very different utilisation on two runs, because API latency fed back into the simulated clock, and the baselines were bound by a loop with no queue at all while K0 had kube-scheduler’s backoff. Phase 2 adds, and tests without a cluster:

  • repeated, interleaved runs reported as mean ± sd with a stability verdict, and a seeded bootstrap screen for differences (with its false-alarm rate measured rather than assumed);
  • the baselines bound through a reproduction of kube-scheduler’s queue (activeQ, backoff, unschedulable pool), so both sides go through the same mechanics;
  • kube-scheduler’s backoff scaled to the replay speed-up, with the residual recorded;
  • a preemption baseline, a startup-delay knob, topology scenarios, and stranded gang GPU-hours in place of a thresholded “deadlock rate”.

Whether five repeats tame the spread on a real control plane is unknown until it runs.

make up        # kind + kwok + the simulated fleet
make bench     # every built configuration, REPEAT=5 times each
make bench-model   # the degenerate baselines in an in-process model; rows labelled "model", never compared with cluster rows