github.com/Zhanyl-tech/ib-slurm-exporter · Go · MIT

built Validated only against synthetic /sys, /proc and cgroup trees built from the kernel and Slurm documentation. Not run on InfiniBand hardware or on a Slurm 26.05 node. The packaging (systemd unit, DaemonSet, container image) has not been run on a real host either.

make demo

No InfiniBand, no cluster, no Linux required: it builds a synthetic /sys, /proc and cgroup tree and serves real metrics off it.

The problem

An illustrative scenario, not a measured incident: a 64-node training run drops from 4,200 to 900 samples/sec. GPUs look busy, Slurm says RUNNING, and a Slurm exporter’s job count, state and runtime do not move. The cause is one HCA retransmitting — packet_seq_err climbing on mlx5_1 — stalling the all-reduce and idling every GPU behind it. Fabric monitoring can see that counter; what it usually cannot tell you is which job on a shared compute node was using that HCA.

Five-hop chain from a Slurm cgroup through a PID and an open uverbs file descriptor to the per-port hardware counter

The join

Neither Slurm nor the HCA knows about the other. By default the bridge is a file descriptor: a process doing RDMA through libibverbs holds one open on a uverbs device.

<scope>/<job>/step_0/user/task_0/cgroup.procs → PID 41001
  → /proc/41001/fd/3 → /dev/infiniband/uverbs0
  → /sys/class/infiniband_verbs/uverbs0/ibdev → "mlx5_0"
  → /sys/class/infiniband/mlx5_0/ports/1/hw_counters/packet_seq_err

The last hop goes through sysfs on purpose. uverbs3 does not have to mean mlx5_3; the numbering is independent, and a host where they happen to match is exactly the host where the shortcut looks correct and is wrong elsewhere.

An opt-in second mode reads queue-pair ownership from the kernel’s RDMA resource tracking instead, which attributes per port and counts kernel RDMA users (NFS/RDMA, Lustre o2ib, IPoIB) that the fd mode is blind to. It has a blind spot of its own, listed below. Its key names come from iproute2’s source; it has not been run on hardware.

What it refuses to do

Port counters are per port. The HCA counts packets; the counters do not record which process sent them. When two jobs share mlx5_1, splitting its port_xmit_data between them is not hard, it is impossible. Exporters that paper over this put a noisy neighbour’s retries on a healthy job, and mislead you exactly when you are debugging with them.

So there are two families. ib_slurm_job_* appears only for a port whose sole visible user is that job, and accumulates only the increase seen while it was the sole user, so a visible neighbour’s traffic never lands in it. ib_slurm_device_* always appears and carries no job label. Anything that leaves the user set incomplete (unreadable /proc, an unresolved SLUID, a failed QP listing) suppresses all job attribution for that sample and is exported as a metric of its own. In the demo mlx5_1 is shared: the synthetic fixture starts it at 48,221 packet_seq_err and keeps it climbing, and that count is still exported, at device level, where it is true.

Slurm 26.05 changed the cgroup layout

From the 26.05 release notes (read 2026-09-27): “cgroup/v2 directory structures are now keyed off of SLUID and not the JobId.” With cgroup.conf’s CgroupJobIdPaths at its default (no), the job directory is the bare SLUID, such as sEKNKTV3WPV500, with no job_ prefix (cgroup.conf). Version 0.1.0 of this exporter assumed job_<SLUID> and would have found no jobs at all on such a node. All three layouts (cgroup v1, v2 by job ID, v2 by SLUID) are now handled; a SLUID is resolved from the job’s slurmstepd process title or from squeue, and anything unresolved is counted, never guessed. That handling is implemented from the documentation and tested on synthetic trees, not on a live 26.05 node.

Limitations

  • Never run on InfiniBand hardware.
  • Ports it can see are shared are never job-attributed, by design; in fd mode, where NCCL opens every active HCA, per-job attribution in practice needs node-exclusive jobs.
  • “Sole user” means sole visible user. In the default fd mode, kernel RDMA users (IPoIB, NFS/RDMA, Lustre o2ib) are invisible, so their traffic can land on a job that looks like the only user; in qp mode, userspace queue pairs created through mlx5 DEVX (UCX’s default on mlx5) are invisible. The README’s ownership table lists each case.
  • No per-QP counter binding yet. Binding the mlx5 retry counters per process would turn suppression into real per-job attribution for that signal; it is the next milestone.
  • mlx5-centric hardware counters; RoCE PFC pause counters are netdev counters and are not collected.
  • Endpoint symptoms only. An HCA counter says something on the path is dropping or reordering, not which link; pair it with a switch-side exporter.