github.com/Zhanyl-tech/cluster-sre-agent · Python · MIT
partial Built and tested: the failure-propagation graph and the read-only command guard. Not built: the agent loop (configurations A–E) and any MCP server, so no diagnosis accuracy has been measured and the results table is empty.
Scored against slurm-rca-bench, and built as five configurations that differ by one layer at a time, because the comparison between them is the finding:
| adds | question | |
|---|---|---|
| A | raw LLM + shell (every command still passes the guard) | within-ablation baseline |
| B | + read-only tools, packaged as an MCP server would expose them | do packaged tools help? |
| C | + the explicit dependency graph | the hypothesis |
| D | + multi-agent specialists | does splitting evidence help? |
| E | + calibrated abstention | does knowing when to stop help? |
If C does not beat B, that is published as the result. A known threat to validity is stated up front: the graph’s measured edges come from the same runs that set two scenarios’ ground truth, so any C-versus-B comparison has to report those scenarios separately.
The graph refuses the folk model
Edges record whether a failure actually propagates, and how strongly, not merely that a dependency exists. Each edge says whether it was measured, documented or inferred:
$ csa stats # run on 2026-09-27
12 edges, 4 measured (33%)
documented 3, inferred 5
An earlier version of this page said 62% measured: three edges were labelled measured with no recorded run behind them. Three hardware edges were also labelled documented with nothing cited. All six have been relabelled.
$ csa causes slurm.scheduler (excerpt)
documented slurm.config 1 hop(s) slurm.config → slurm.scheduler
documented slurm.slurmctld 1 hop(s) slurm.slurmctld → slurm.scheduler
inferred storage.state_save 2 hop(s) storage.state_save → slurm.slurmctld → slurm.scheduler
only through an untested failure mode: a StateSaveLocation *stall* (writes block instead of failing), which S06's caveats name as a candidate for blocking slurmctld; S06 measured the error mode only
ruled out at halts by measurement, within this graph:
slurm.slurmdbd slurm.slurmdbd → slurm.scheduler only degrades
db.mysql slurm.slurmdbd → slurm.scheduler only degrades
storage.shared_fs slurm.slurmdbd → slurm.scheduler only degrades
not covered by those measurements:
slurm.slurmdbd → slurm.scheduler:
- accounting-queue saturation: the DBD agent queue peaked at 6, while slurm.conf documents a MaxDBDMsgs floor of 10000
- AccountingStorageEnforce=associations or limits (S08 found the benchmark cluster set to none)
- stalls longer than the roughly fifteen minutes observed
Being able to say “I checked the accounting path and, in the regime that was
measured, it cannot produce this symptom” is worth as much as naming the
cause, and the last section says exactly where each measurement stops. A
measurement covers only the failure mode it exercised: the benchmark measured a
StateSaveLocation that errors, which only degrades the controller, so
storage.state_save stays on the list as an inferred cause through the
untested stall mode rather than being ruled out. (An earlier version of the
graph, and of this page, listed it as ruled out by measurement.)
Severity composes along a path and does not compose transitively. The first
version of the traversal got this wrong and a test caught it: every arrow in
mysql → slurmdbd → slurmctld → scheduler exists, so a naive walk concludes the
database can stop scheduling, reintroducing the model the benchmark refuted. A
path is only as strong as its weakest link.
Read-only is a control, not a request
A prompt saying “only use read-only commands” fails open. The guard every
execution path calls is an allowlist: eight read tools, named bare and run
by absolute path from trusted directories; every option spelled exactly as the
tool’s own parser spells it (so --res is not taken for --reset); full-word
read subcommands only. The first version was a denylist, and an audit got
scontrol -o shutdown, scontrol upd PartitionName=debug State=DOWN (scontrol
expands upd to update), sacctmgr -i delete user x, sdiag -r and a bare
scontrol (which reads commands from stdin) through it. The tests now cover every
scontrol and sacctmgr command on the man pages and every abbreviation of
every write, and CI fails if any listed write is allowed.
Model
Every configuration is to run the same model, pinned as claude-opus-5 for
reproducibility, so the ablation isolates architecture rather than model
choice. Configuration A is a baseline within this ablation, not comparable to
published numbers from other harnesses.