github.com/Zhanyl-tech/epilog-gpu-validator · Go · MIT
built
Exercised against a simulator, synthetic nvidia-smi output and
fake scontrol/nvidia-smi binaries. No GPU and
no Slurm cluster were used. Where it relies on NVIDIA or Slurm
behaviour it cites the documentation, and the README lists the assumptions
still unverified. Report-only is the default.
make scenarios
Every case runs the real decision code with --enforce against a simulated
nvidia-smi and a stand-in scontrol that only records calls. Output of
--scenario-table on 2026-09-27:
CASE STATUS WORST EXIT DETAIL
----------------------------------------------------------------------------------------------------------------------
healthy ok unknown 0 pcie-gen:gpu0,gpu1
pcie-degraded ok unknown 0 pcie-gen:gpu0,gpu1 pcie-width:gpu0
remap-pending ok transient 0 row-remap-pending:gpu0 row-remap-uncorrectable:gpu0
thermal ok transient 0 throttle:gpu0
ecc-na ok unknown 0 pcie-gen:gpu0,gpu1 unreadable:gpu0
no-persistence ok unknown 0 ecc-volatile-unreliable:gpu0 pcie-gen:gpu0,gpu1
no-driver not-checked unknown 0 query-failed:node
hw-slowdown drained degraded 1 epilog-gpu-validator degraded: throttle:gpu0
leaked-memory drained degraded 1 epilog-gpu-validator degraded: leaked-memory:gpu0
ecc drained fatal 1 epilog-gpu-validator fatal: ecc-uncorrectable:gpu0
remap-failure drained fatal 1 epilog-gpu-validator fatal: row-remap-failure:gpu0
off-bus drained fatal 1 epilog-gpu-validator fatal: nvidia-smi-exit-15:node
leaked-memory --shared-gpus ok unknown 0 leaked-memory:gpu0 pcie-gen:gpu0,gpu1
pcie-degraded --drain-on-pcie-width drained degraded 1 epilog-gpu-validator degraded: pcie-width:gpu0
ecc, minors reversed not-checked unknown 0 gpu-number-ambiguous:slurm-gpu0,slurm-gpu1
ecc, minors reversed --gpu-numbering nvml drained fatal 1 epilog-gpu-validator fatal: ecc-uncorrectable:gpu0
ecc, minors reversed --gpu-numbering minor ok unknown 0 pcie-gen:gpu2,gpu3
ecc, minors reversed, env_uuid drained fatal 1 epilog-gpu-validator fatal: ecc-uncorrectable:gpu0
job GPU missing from output partially-checked unknown 0 not-in-output:slurm-gpu1 pcie-gen:gpu0
ecc, scontrol group-writable drain-failed fatal 1 epilog-gpu-validator fatal: ecc-uncorrectable:gpu0
no GPUs in env no-gpus ok 0 -
GPU vars disagree not-checked unknown 0 gpu-set-ambiguous:node
nvidia-smi absent config-error unknown 0 --nvidia-smi: /usr/bin/nvidia-smi: no such file or directory
flag typo --budget 20 config-error unknown 0 bad command line: invalid value "20" for flag -budget: parse error
unknown flag config-error unknown 0 bad command line: flag provided but not defined: -no-such-flag
These rows show how the simulator is modelled, not hardware results: the
simulated nvidia-smi output was written from NVIDIA’s documentation, not
captured from a GPU. The simulated job holds Slurm GPUs 0 and 1, and every
fault is placed on nvidia-smi’s gpu0; the minors reversed rows number the
device files in reverse PCI order to show what each --gpu-numbering value
(below) does with that. A test (TestREADMEScenarioTableIsCurrent) compares the
README’s copy of this table with the command’s output byte for byte, so it
cannot drift silently.
The problem
A GPU develops a fault mid-job. The job fails or, worse, silently returns wrong numbers. Slurm marks the node idle. The next job lands on the same card and fails the same way. Slurm provides the hooks for a health check but not the check: the Epilog runs on every job completion, on the node, as root.
Why it is a dangerous tool to write
Slurm drains the node when the Epilog exits non-zero (“If the Epilog fails (returns a non-zero exit code), this will result in the node being set to a DRAIN state”, prolog/epilog guide). So a false positive does not produce a bad metric; it removes a working node, and a fleet-wide cause removes every node, one job completion at a time.
That is not hypothetical. An audit of version 0.1.0 found two fleet-wide
problems in it. NVIDIA documents that an idle GPU’s PCIe link “may be reduced
when the GPU is not in use”, and 0.1.0 drained on a generation below maximum,
so on hardware that does this every node would drain after its next job (the
audit reproduced it with a synthetic idle row, not on a GPU). And the Epilog
runs with no PATH, so 0.1.0 never found nvidia-smi and silently checked
nothing. Both are fixed. Three rules follow:
- Never drain on ignorance. A missing, timed-out or unparseable
nvidia-smi, or a field reported as[N/A], is a monitoring failure: exit 0, loudly logged. Only four documentednvidia-smiexit codes (8, 10, 14, 15) count as hardware evidence. - Separate persistent faults from conditions that clear. Uncorrectable ECC since the last driver load and a row-remapping failure are fatal; a hard thermal slowdown or power brake on an idle GPU, or memory still held after teardown, is degraded; thermal throttling, a pending row remap and rows already remapped are transient, and so is uncorrectable ECC seen only in the lifetime count, but only when persistence mode is on (otherwise it is unknown); a PCIe link below maximum is unknown, because NVIDIA documents that it “may be reduced when the GPU is not in use”.
- Only one exit is non-zero: a degraded or fatal finding, with
--enforce, on a real run. Misconfiguration, an ambiguous GPU set and a failed query without one of those four exit codes all exit 0, and all log at error level.
Volatile versus aggregate ECC is the distinction worth labouring: volatile counters reset when the driver unloads, and the driver unloads when no clients remain unless persistence mode is on, so a zero volatile count at Epilog time proves nothing without it, and the tool says so.
Only the job’s own GPUs
On a shared node, checking every GPU means a neighbour’s faulty card drains the
node for a job that never touched it. So the tool checks only the GPUs the
Epilog environment names (CUDA_VISIBLE_DEVICES, SLURM_JOB_GPUS,
GPU_DEVICE_ORDINAL; if more than one is set they must agree), and checks
nothing it cannot tie to the job without guessing.
Those variables hold Slurm’s own GPU index (established from Slurm’s source;
the documentation does not say), and which GPU that index means depends on
gres.conf:
nvidia-smi’s number with AutoDetect=nvml, the N in /dev/nvidiaN
without it. --gpu-numbering says which applies. The default, auto, checks
a numbered GPU only where the two agree on this node (device minors in PCI bus
order) and otherwise reports it unchecked; nvml, minor and uuid state the
mapping, and --check-config fails on a node where auto would check nothing.
With gres.conf Flags=env_uuid (Slurm 26.05 and later) the job’s GPUs arrive
as UUIDs, which name one device whatever the numbering. The one node-wide
signal is a documented hardware-fault exit code from nvidia-smi, such as a
GPU that has fallen off the bus.
What it deliberately does not do
- No load test. An active test catches faults a passive query cannot, but the Epilog’s time budget is short, and a killed Epilog drains the node for a check that never concluded. DCGM diagnostics or a periodic drain-and-test job are the right home for that.
- No Xid parsing, no MIG, no history. Each run is one snapshot; MIG devices are reported and not checked.
- Never un-drains. Bringing a node back is a human decision.
- Under Slinky (slurmd in a container) it is untested; the README lists what must hold inside the container first.
The set
One of three fleet tools with one rule, never act on absent evidence: gpu-reaper treats a telemetry gap as a collector fault rather than idleness, ib-slurm-exporter refuses to attribute a port it can see is shared, and this exits clean when it cannot see the hardware.