GPU scheduling and fleet health: what was measured, and on what.
I'm Zhanyl Abdybaeva, an HPC and ML infrastructure
engineer. I build the systems that allocate scarce, expensive,
heterogeneous compute, and the benchmarks that check whether they work.
Ten years of production infrastructure ·
MS CS (machine learning), Georgia Tech · CQF · NVIDIA NCP-AIO
The public work here is independent (coursework, CQF projects and my
own research, run on kind clusters, simulators and synthetic data) and
covers three things: controlled evaluation of GPU
schedulers across Slurm and Kubernetes (including Slurm on
Kubernetes via Slinky), GPU fleet-health automation that
refuses to act on missing evidence, and a habit of publishing
what was measured separately from what was asserted,
negative results and corrections included.
Every project page states what is built, partial or planned.
partialSlurm model built · Kubernetes K0 + three degenerate baselines measured once (D-preempt model-only) · Kueue, Volcano, KAI planned
The same largest-first schedule reads 0.0% or 55.6% GPU fragmentation depending on the definition used. A fragmentation claim that does not state its definition cannot be reproduced. (kind + kwok control plane, Phase 1, one run.)
partialbring-up built · key rotation blocked, fails safe · Argo CD not wired
kubectl rollout restart silently skips slurmd pods owned by Slinky's NodeSet custom resource. The rotation script hashes the key inside every slurmd pod and rolls back when it does not match the Secret.
builttested against simulators and synthetic trees, not on GPU or InfiniBand hardware
Three tools, one rule: never act on missing evidence. An nvidia-smi query that fails without a documented hardware-fault exit code exits 0 instead of draining; a gap in telemetry is a collector fault, not an idle GPU; a port with more than one visible user is never attributed to a job, and the project page lists the users each mode cannot see.
The incident benchmark's flagship scenario encoded a widely repeated chain: a stalled accounting database halts scheduling. On a CPU-only Docker Compose test cluster (Slurm 25.11.4) it did not, so the scenario was rewritten. Corrections to this site are listed too.
built code and tests exist ·
partial some named parts are not built ·
planned not built.
Nothing on this map has run on a production GPU cluster; each project page says what it was tested against.
Structured sparsity aligned with GPU attention kernels: LServe’s unified serving stack for prefill and decode, and SampleAttention’s empirical patterns plus CRA as a runtime quality floor.
A walkthrough of PagedAttention, the KV-cache memory manager from the vLLM paper (Kwon et al., SOSP 2023), and why paging matters for serving many concurrent LLM sessions.