Slurm

Drain a node for a persistently faulty GPU — and never for a transient one.

Slurm on Kubernetes from nothing in one command — and the auth-key rotation nobody wants to test in production.

Correlate InfiniBand and RoCE counters with the Slurm job that owns them, so a slow collective can be traced to the fabric.

Find wasted GPU allocations on a Slurm cluster and escalate them through alert, drain, and cancel.

Test Slurm priority and backfill policy against a job trace before it reaches a live controller.

Three versions, two config renames that break on restart, and one cgroup change that quietly breaks every job-attribution tool you own.

I built a simulator to tune Slurm priority weights. It kept telling me the weights were not the problem.