HPC & GPU infrastructure · Slurm & Kubernetes · measured versus asserted

GPU scheduling and fleet health: what was measured, and on what.

I'm Zhanyl Abdybaeva, an HPC and ML infrastructure engineer. I build the systems that allocate scarce, expensive, heterogeneous compute, and the benchmarks that check whether they work.

Ten years of production infrastructure · MS CS (machine learning), Georgia Tech · CQF · NVIDIA NCP-AIO

The public work here is independent (coursework, CQF projects and my own research, run on kind clusters, simulators and synthetic data) and covers three things: controlled evaluation of GPU schedulers across Slurm and Kubernetes (including Slurm on Kubernetes via Slinky), GPU fleet-health automation that refuses to act on missing evidence, and a habit of publishing what was measured separately from what was asserted, negative results and corrections included. Every project page states what is built, partial or planned.

cat ./evidence

Scheduler evaluation · Slurm × Kubernetes

partial Slurm model built · Kubernetes K0 + three degenerate baselines measured once (D-preempt model-only) · Kueue, Volcano, KAI planned

The same largest-first schedule reads 0.0% or 55.6% GPU fragmentation depending on the definition used. A fragmentation claim that does not state its definition cannot be reproduced. (kind + kwok control plane, Phase 1, one run.)

Slurm on Kubernetes · Slinky

partial bring-up built · key rotation blocked, fails safe · Argo CD not wired

kubectl rollout restart silently skips slurmd pods owned by Slinky's NodeSet custom resource. The rotation script hashes the key inside every slurmd pod and rolls back when it does not match the Secret.

GPU fleet health · Go

built tested against simulators and synthetic trees, not on GPU or InfiniBand hardware

Three tools, one rule: never act on missing evidence. An nvidia-smi query that fails without a documented hardware-fault exit code exits 0 instead of draining; a gap in telemetry is a collector fault, not an idle GPU; a port with more than one visible user is never attributed to a job, and the project page lists the users each mode cannot see.

Measured versus asserted

partial 10 scenarios · 2 causal chains measured · no agent scored yet

The incident benchmark's flagship scenario encoded a widely repeated chain: a stalled accounting database halts scheduling. On a CPU-only Docker Compose test cluster (Slurm 25.11.4) it did not, so the scenario was rewritten. Corrections to this site are listed too.

System map with a status on every box. Top band, measurement: slurm-rca-bench (partial: 10 scenarios, 2 causal chains measured, no agent scored) and k8s-gpu-scheduler-lab (partial: K0 and four degenerate baselines built; the Phase 2 harness is built but has not run on a cluster). Middle left, the Slurm control plane: slurm-scheduler-lab (built: a simulator, no live controller behind its numbers), cluster-sre-agent (partial: dependency graph and read-only guard built, no MCP server of its own, agent configurations planned), slurm-mcp (built: a read-only MCP server, fixture-tested), and epilog-gpu-validator (built, simulator-tested; drains only on fault evidence, never on a failed query without it). Middle right, the Kubernetes control plane: slinky-gitops (partial: Slurm on kind built, auth-key rotation blocked and rolls back safely, Argo CD not wired) and the schedulers under comparison, Kueue, Volcano and NVIDIA KAI, all planned. Bottom band, the GPU fleet: gpu-reaper and ib-slurm-exporter (built, tested on simulated telemetry and synthetic trees, not on hardware; the exporter attributes counters to a job only where it is the sole visible user of the port). Built means code and tests exist; nothing here has run on a production GPU cluster.
built code and tests exist · partial some named parts are not built · planned not built. Nothing on this map has run on a production GPU cluster; each project page says what it was tested against.

Last published July 26, 2026

tail ./lab-notes

All lab notes →

ls ./writing

All deep dives →