github.com/Zhanyl-tech/slurm-scheduler-lab · Python · MIT
built
A simulator. Its semantics come from Slurm's documentation and source (pinned
to a commit in the README); nothing here has been replayed against a real
slurmctld, and every number on this page is from its synthetic
workload unless it says otherwise.
Changing PriorityWeightFairshare on a production cluster is a slow, expensive
experiment: the feedback loop is days long, the blast radius is everyone’s
queue, and the only rollback signal is people complaining. This runs the same
policy against a few hundred jobs in about a second, and reads the settings
straight out of the slurm.conf you are about to deploy.
What it models
- Multifactor priority, the
priority/multifactorformula, with each factor normalised to [0, 1] so the weights alone decide what the queue optimises for. Integer priorities, as Slurm stores them. - Fairshare, both of Slurm’s algorithms. Fair Tree by default, because it
is what an unconfigured controller runs; classic
2^(-U/S)withPriorityFlags=NO_FAIR_TREE. Usage accrues and decays onPriorityCalcPeriodticks, so priorities lag the way they do on a real controller. - Backfill, two ways. The default is the shape of Slurm’s
sched/backfill: a main scheduler that stops at its first blocked job, plus a backfill cycle everybf_intervalthat plans every pending job into a per-node timeline and starts a job only if it moves no higher-priority job’s planned start, bounded bybf_max_job_test,bf_windowandbf_resolution. Textbook EASY backfill (protect only the first blocked job) is kept as an option. - Preemption, off by default as in Slurm:
preempt/partition_prioandpreempt/qos,CANCELandREQUEUE, andGraceTime, with work lost and grace-locked CPU- and GPU-hours reported. - The scheduler never reads a job’s true runtime, only its requested time
limit, and a test enforces that across whole simulations. An
saccttrace keeps its recordedTimelimit; it is never replaced byElapsed. - The Slurm side of a cross-substrate comparison (S0). It can replay a trace from k8s-gpu-scheduler-lab on the same heterogeneous fleet and score it under that lab’s metric definitions, including a shared “Definition C” fragmentation function checked against a common golden vector. As a model only; the README lists what is and is not comparable, starting with the fact that Slurm allocates a gang atomically.
Settings the model does not implement are printed as warnings rather than silently ignored.
Point it at your cluster
sacct -a -X --parsable2 --starttime=now-30days \
--format=JobID,User,Account,Submit,Elapsed,Timelimit,NNodes,ReqCPUS,ReqTRES \
> trace.txt
schedlab --slurm-conf /etc/slurm/slurm.conf --sacct trace.txt --nodes 64 --cpus 32 --gpus 4
No controller is touched. --sched-params tries a SchedulerParameters
change without editing the config, and --sweep walks one priority weight.
What it shows, and how much to trust it
On one synthetic seed the effect of backfill looks dramatic:
$ schedlab --compare-backfill --jobs 300 --seed 5 # 16 nodes × 8 CPU / 2 GPU
backfill OFF cpu utilization 72.3 % mean wait 1896.7 min
backfill ON cpu utilization 83.0 % mean wait 396.1 min
One seed is an anecdote, and this project learned that the hard way: an earlier write-up drew “priority weights barely matter” from a single seed of version 0.1, and it did not survive twenty (correction). Over seeds 0–19 of the current model, run on 2026-09-27:
| median | range | |
|---|---|---|
| mean wait, backfill off ÷ on | 3.84× | 2.17–5.71× |
| CPU utilisation gain from backfill | +16.4 points | 5.1–28.8 |
mean wait, largest ÷ smallest across PriorityWeightJobsize 0 / 1k / 10k / 100k | 3.18× | 2.15–3.87× |
Backfill cut mean wait on every seed; the job-size weight moved it by a
comparable factor, and more than backfill did on 6 of the 20. Each row is
schedlab --compare-backfill --jobs 300 --seed s and
schedlab --sweep jobsize --jobs 300 --seed s for each seed, read from the
text report. Exact time limits (--time-limit-model exact, over seeds 0–9)
raised mean CPU utilisation from 84.2% to 86.3%, higher on 7 of the 10 seeds,
and mean wait from 489.1 to 656.6 minutes, higher on 9 of 10
(schedlab --jobs 300 --seeds 10, with and without that flag): accurate
requests bought some throughput and cost waiting on this workload.
--seeds N prints the spread for any comparison, and the README runs every
claim it makes through it.
Scope
Deliberately not a Slurm reimplementation. First-fit node selection with no
topology-aware placement; a flat, one-level account tree; no job arrays,
heterogeneous jobs, licences, reservations or gang time-slicing; Slurm’s own
wall-clock costs (bf_max_time, RPC load) are not modelled. Good for comparing
policies against each other on the same workload, not for predicting absolute
wait times on your cluster. The README lists every simplification and the
source line each modelled rule comes from.