[{"content":"github.com/Zhanyl-tech/epilog-gpu-validator · Go · MIT\nmake scenarios Every fault case against a synthetic GPU. No NVIDIA hardware required.\nSCENARIO SEVERITY DRAIN EXIT healthy ok no 0 remap-pending transient no 0 thermal transient no 0 pcie-degraded degraded yes 1 hw-slowdown degraded yes 1 ecc fatal yes 1 remap-failure fatal yes 1 missing n/a no 0 ← query failed, safe no-op The problem A GPU develops a fault mid-job. The job fails, or worse, silently returns wrong numbers. Slurm marks the node idle. The next job lands on the same card and fails the same way. Then the next.\nNothing in Slurm looks at GPU health between jobs. Epilog is the hook that could — it runs on every completion, on the node, as root, while nothing else is touching the hardware.\nWhy it\u0026rsquo;s a dangerous tool to write Slurm drains the node when Epilog exits non-zero.\nA false positive doesn\u0026rsquo;t produce a bad metric. It removes a working node. And if the cause is fleet-wide — a driver bug, a monitoring gap, a heatwave — it removes every node, one job completion at a time, faster than anyone can react.\nTwo rules follow, and everything else is detail.\nNever drain on ignorance. If nvidia-smi is missing, times out, or returns something unparseable, that\u0026rsquo;s a monitoring failure. Draining on it converts a broken health check into a cluster-wide outage. That\u0026rsquo;s the missing row above, and it\u0026rsquo;s the most important line in the test suite.\nSeparate persistent faults from transient conditions. A card thermal-throttling on a hot afternoon is the hardware protecting itself — draining for it means draining the row every time the CRAC unit hiccups. A card running x8 on an x16 slot is a different animal: it passes every functional test, halves host-to-device bandwidth, and poisons every collective it joins. Nobody notices for months.\nThe distinction I found most worth labouring is volatile versus aggregate ECC. Uncorrectable errors since boot mean the card is failing right now — the job that just finished may have produced wrong numbers. Lifetime errors on a card that\u0026rsquo;s been clean since its last reset justify nothing.\nOnly the job\u0026rsquo;s own GPUs On a shared node, checking everything means a neighbour\u0026rsquo;s faulty card drains the node for a job that never touched it. It reads SLURM_JOB_GPUS and checks only those — and if that set can\u0026rsquo;t be determined, it checks nothing and exits 0.\nWorth knowing: depending on Slurm version and GresTypes, those variables can contain UUIDs rather than ordinals. Those are skipped rather than guessed at.\nWhat\u0026rsquo;s deliberately missing No dcgmproftester. An active memory-bandwidth test would catch faults that only appear under load, and the original plan called for one. But it takes tens of seconds and Epilog doesn\u0026rsquo;t have that budget — if Slurm kills the check mid-run, the node can be marked down for a check that never reached a conclusion. That\u0026rsquo;s the opposite of the point. A periodic drain-and-test job is the right home for it.\nAlso no XID scraping (needs privileges Epilog doesn\u0026rsquo;t reliably have), no MIG awareness, and it never un-drains — bringing a node back is a human decision.\nThe set This is the fourth of four. Together they cover the lifecycle of a GPU allocation: gpu-reaper catches waste while a job runs, ib-slurm-exporter attributes fabric problems to the job causing them, this validates the hardware between jobs, and slinky-gitops runs the whole thing on Kubernetes.\nThe same principle runs through all four: never act on absent evidence. gpu-reaper treats a sampling gap as a collector fault rather than idleness. ib-slurm-exporter refuses to attribute a shared device. This exits clean when it can\u0026rsquo;t see the hardware. In every case the failure mode of guessing is worse than the cost of not knowing.\n","permalink":"https://zhanyl-tech.github.io/projects/epilog-gpu-validator/","summary":"Drain a node for a persistently faulty GPU — and never for a transient one.","title":"Epilog GPU Validator"},{"content":"Status: 🟡 In Progress — Deadline August 2026\nStack: vLLM · TensorRT-LLM · LangGraph · MCP · gRPC · Kubernetes · CUDA\nWhat This Is The infrastructure layer for deploying agentic AI systems in production. Not the agents themselves — the platform that makes them reliable, fast, and observable at scale.\nThis sits at the emerging boundary between inference serving (get tokens out fast) and agentic orchestration (make reliable multi-step decisions). Most teams treat these separately. This project builds the bridge.\nProblem Statement Deploying a single LLM for generation is a solved problem. Deploying an agent that:\nMakes tool calls with sub-second decision latency Maintains state across multi-step reasoning chains Fails gracefully when tools return unexpected results Runs concurrently across hundreds of simultaneous sessions Is observable enough that you can debug production failures \u0026hellip;is not solved. This project is building that infrastructure.\nArchitecture ┌─────────────────────────────────────────────────────────┐ │ Agent Request Layer │ │ gRPC API · Session Management │ └─────────────────────┬───────────────────────────────────┘ │ ┌─────────────────────▼───────────────────────────────────┐ │ LangGraph Orchestration │ │ State Machine · Tool Registry · Retry Logic │ └──────────┬──────────────────────────┬───────────────────┘ │ │ ┌──────────▼──────────┐ ┌───────────▼───────────────────┐ │ vLLM Inference │ │ MCP Tool Server │ │ (token generation) │ │ (structured tool execution) │ │ Multi-GPU · KV │ │ Yahoo Finance · Code Exec · │ │ Cache Paging │ │ Database · Custom APIs │ └──────────┬──────────┘ └───────────────────────────────┘ │ ┌──────────▼──────────────────────────────────────────────┐ │ Observability Layer │ │ Prometheus · Grafana · Distributed Tracing │ └─────────────────────────────────────────────────────────┘ Key Technical Challenges 1. KV-Cache Management for Long Agent Sessions Multi-step reasoning chains can span dozens of turns. Standard KV-cache strategies waste memory on completed reasoning steps. Solution: prefix caching + aggressive eviction for completed tool call branches.\n2. Concurrent Session Isolation 100 simultaneous agent sessions shouldn\u0026rsquo;t interfere with each other\u0026rsquo;s state. Building session-isolated execution contexts with shared inference backend.\n3. Tool Call Reliability When an MCP tool returns an error, the agent needs to decide: retry, use fallback, or fail cleanly? Building a typed error taxonomy so agents can reason about failures.\n4. Latency Budget Management Each reasoning step has a time budget. If tool calls exceed it, the agent needs to degrade gracefully (use cached results, skip tool, etc.) rather than hang.\nBenchmarks (Ongoing) Metric Target Current Tool call round-trip p50 \u0026lt; 200ms — Tool call round-trip p99 \u0026lt; 500ms — Concurrent sessions 100+ — Agent decision latency p50 \u0026lt; 1s — Throughput 500+ req/s — Will update as benchmarks complete.\nMilestones M1 (March 2026): Single-agent session, vLLM + LangGraph + 3 MCP tools working end-to-end M2 (May 2026): Concurrent session handling, observability stack, baseline benchmarks M3 (July 2026): Production hardening — error handling, graceful degradation, load testing M4 (August 2026): Full benchmark suite, architecture writeup, public repo Related Posts Posts will appear here as they\u0026rsquo;re published.\nGitHub: Available August 2026\n","permalink":"https://zhanyl-tech.github.io/projects/agentic-ml-inference-infrastructure/","summary":"Building the infrastructure layer for autonomous AI systems in production — serving, orchestration, and tooling for LLM-powered agents at scale.","title":"Agentic ML Inference Infrastructure"},{"content":"github.com/Zhanyl-tech/slinky-gitops · MIT\nmake up Three-node KinD cluster → cert-manager → Slinky operator → a Slurm cluster that registers a compute node and runs jobs. Verified end to end on Apple Silicon: the Slinky images are multi-arch, which is not obvious and is the first thing that stops most people.\n$ make job slinky-0 $ make status all* up infinite 1 idle slinky-0 NodeName=slinky-0 Arch=aarch64 State=IDLE+DYNAMIC_NORM Every MUNGE rotation guide describes a component that isn\u0026rsquo;t installed I set out to build a MUNGE key-rotation sidecar. Then I deployed the thing and checked:\n$ scontrol show config | grep -i auth AuthType = auth/slurm CredType = cred/slurm AuthAltTypes = auth/jwt $ pgrep munged (nothing) Slurm 23.11 introduced auth/slurm, an internal plugin that replaces MUNGE with a shared key file. Slinky uses it. MUNGE is not installed.\nSo the hazard is unchanged — one shared secret, every daemon, no atomic switchover — but the mechanism, the secret names, and the restart procedure in every tutorial are wrong. The secrets are slurm-auth-slurm (slurm.key) and slurm-auth-jwt (jwt.key).\nBuilding from the brief instead of from the running cluster would have produced a polished tool for a component that isn\u0026rsquo;t there.\nRotation, done carefully There\u0026rsquo;s no atomic moment. Between writing the new key and every daemon reloading it, some hold the old key and some the new, and those two sets cannot authenticate to each other. auth/slurm has no key versioning and no grace period.\nSo: refuse to start unless the cluster is healthy and actually on auth/slurm; drain first, because running jobs are what a failed rotation destroys; keep the previous key so rollback is one command; verify end to end with a call that round-trips through the auth plugin; roll back automatically if that fails.\nFour bugs I only found by running it The auth secrets are immutable. kubectl patch is rejected outright — field is immutable when immutable is set. The only path is delete-and-recreate, carrying the labels so the operator and Helm still recognise it.\nThe first version failed exactly there and left the cluster drained. Which is worse than not having rotated. Failure now resumes nodes on every exit path.\nDelete-and-recreate silently drops immutable: true. You get a working, mutable secret. Everything keeps running, so nothing tells you the posture just weakened. That\u0026rsquo;s the one I\u0026rsquo;d have shipped without noticing — it\u0026rsquo;s invisible unless you go looking. The flag is now captured before the delete and restored after, verified by rotating twice and re-checking.\nThe verification step verified nothing. This is the one worth keeping.\nCI failed with the rotation printing a green tick on every step, then Rotation complete, and then:\nsrun: Required node not available (down, drained or reserved) Rotation can only break one thing — the trust between slurmctld and slurmd, because that\u0026rsquo;s what the rotated key authenticates. My check ran sinfo inside the controller pod and looked at the exit code, which never touches that relationship: slurmctld answers a local client whether or not a single compute node ever came back. It would have passed against a cluster with zero nodes.\nPulling that thread found four defects, all the same bug — a check that can\u0026rsquo;t fail:\nCheck Why it passed anyway sinfo exits 0 Never leaves the controller pod No * on any node state Raced the restart; passed 130 ms after it grep -E '^(idle|mix|alloc)' Also matches idle* — a node that can\u0026rsquo;t be reached, i.e. the failure itself Wait for replacement pods No success flag, so a timeout fell through to a green tick And one defect in the deployment rather than my script:\n$ kubectl get pod slurm-worker-slinky-0 -o jsonpath=\u0026#39;{.metadata.ownerReferences[*].kind}\u0026#39; NodeSet kubectl rollout restart only understands built-in workload kinds. NodeSet is a CRD, so the command I trusted to cycle every daemon silently skipped the one on the far side of the boundary I was rotating. The controller took the new key in seconds; slurmd kept the old one.\nThe part I couldn\u0026rsquo;t fix With all of that corrected, the rotation still fails — and now says so. On a clean cluster, with the Secret holding a new key and stable for five minutes, a slurmd pod deleted and recreated from scratch came up mounting the previous key:\nsecret c5016281… slurmd pod created 02:28:25 8cbda076… ← pre-rotation key slurmctld c5016281… ← correct Slinky ships the auth Secret immutable: true, so its data can\u0026rsquo;t be patched and delete-and-recreate is the only route — after which the node\u0026rsquo;s kubelet keeps serving its cached copy to newly created pods. What I ruled out, because being precise about the boundary of what I know is the whole point:\nNot the operator rewriting it — one value, resourceVersion unchanged, for 90s. Not immutability — recreating the replacement as mutable behaves identically. Not universal — the same delete-and-recreate against a Secret that node had never cached propagates instantly. That control test made me dismiss the whole thing too early, until I realised it didn\u0026rsquo;t reproduce the one condition that matters: a pod holding the Secret continuously across the swap. I can reproduce it on demand and I can\u0026rsquo;t attribute it to a specific kubelet code path, so the repo doesn\u0026rsquo;t claim one.\nWhat the tool does about it is the actual deliverable: it hashes slurm.key inside every slurmd pod, compares it to the Secret, and rolls back and exits non-zero on mismatch. CI asserts that safety property rather than a success it can\u0026rsquo;t have — make rotate must fail, and the cluster must still run a job afterwards.\nThe general form is worth more than any single bug: a check that doesn\u0026rsquo;t cross the boundary you might have broken will pass no matter what you broke. Green ticks on a dead cluster are worse than a red one, because they stop you looking.\nNone of this is something I\u0026rsquo;d have predicted from reading the docs.\nHonest scope The repo says this plainly and so should this page: ArgoCD is not wired up yet, so \u0026ldquo;GitOps\u0026rdquo; in the name is currently aspirational. What exists is declarative and reproducible, applied by Make rather than reconciled by a controller. One nodeset, one replica — enough to prove registration and job execution; autoscaling profiles and multi-nodeset scheduling aren\u0026rsquo;t built. No login node; jobs are submitted from the controller pod.\nAlso worth noting for anyone mid-upgrade: Slinky v1.2 ships Slurm 26.05. The Kubernetes path is already a release ahead of the 25.11 most on-prem clusters are planning for.\n","permalink":"https://zhanyl-tech.github.io/projects/slinky-gitops/","summary":"Slurm on Kubernetes from nothing in one command — and the auth-key rotation nobody wants to test in production.","title":"Slinky GitOps"},{"content":"Status: 🟡 In Progress — Deadline August 2026\nStack: PyTorch FSDP · DDP · Ray · SLURM · Kubernetes · NCCL · Mixed Precision\nWhat This Is Production distributed training infrastructure for training large-scale ML models across 128 GPUs. Covers the full stack: job orchestration, efficient parallelism strategies, fault tolerance, and cost optimization.\nThis is the infrastructure that makes it practical to train models that don\u0026rsquo;t fit on a single GPU — or even a single node.\nProblem Statement Training a 7B parameter model on a single A100 is straightforward. Training a 70B+ model across multiple nodes with:\nEfficient gradient synchronization (minimize communication overhead) Fault tolerance (one GPU failure shouldn\u0026rsquo;t kill a 10-hour run) Optimal parallelism strategy (when to use FSDP vs DDP vs tensor parallel) Cost control (minimize GPU-hours without sacrificing convergence) Reproducibility (same seeds, same results, every run) \u0026hellip;requires purpose-built infrastructure. This project builds it.\nArchitecture ┌──────────────────────────────────────────────────────────────┐ │ Job Scheduler Layer │ │ SLURM / Ray · Job Queue · Priority │ └──────────────────────┬───────────────────────────────────────┘ │ ┌──────────────────────▼───────────────────────────────────────┐ │ Distributed Training Layer │ ├─────────────────┬────────────────────┬────────────────────── ┤ │ FSDP Sharding │ Gradient Ckpt │ Mixed Precision │ │ (param shards │ (activation │ (BF16 forward, │ │ across ranks) │ recomputation) │ FP32 optimizer) │ └─────────────────┴────────────────────┴───────────────────────┘ │ ┌──────────────────────▼───────────────────────────────────────┐ │ Communication Layer │ │ NCCL · AllReduce · AllGather · Ring Topology │ └──────────────────────┬───────────────────────────────────────┘ │ ┌──────────────────────▼───────────────────────────────────────┐ │ Hardware Layer │ │ 128x A100 80GB · NVLink · InfiniBand HDR │ └──────────────────────────────────────────────────────────────┘ Key Technical Decisions FSDP vs DDP: When to Use What The decision isn\u0026rsquo;t obvious and most documentation doesn\u0026rsquo;t explain the tradeoffs clearly:\nUse DDP when: Model fits in GPU memory with a reasonable batch size. DDP is simpler, faster (no sharding communication overhead), and easier to debug.\nUse FSDP when: Model doesn\u0026rsquo;t fit in single GPU memory, or you need to scale across many nodes efficiently. FSDP shards parameters, gradients, AND optimizer state — enabling training of models 4-8x larger than DDP allows.\nThe grey zone: 7B-13B models often fit in DDP if you\u0026rsquo;re aggressive about gradient checkpointing and mixed precision. Worth benchmarking before committing to FSDP complexity.\nGradient Checkpointing Strategy Naive gradient checkpointing (recompute everything) reduces memory usage by ~60% but increases compute by ~30%. Selective checkpointing (recompute only expensive operations) gets most of the memory savings with less compute overhead.\nImplementing an activation memory profiler to identify which layers benefit most from recomputation.\nFault Tolerance at Scale At 128 GPUs, hardware failures are a matter of when, not if. Building:\nAsync checkpoint saves every N steps (configurable) Failure detection with automatic job restart from last checkpoint Degraded-mode training (run on N-1 GPUs if one fails, sync optimizer state) Benchmarks (Planned) Model Size GPUs Batch Size Throughput GPU Utilization 7B 8 32 — — 13B 16 16 — — 70B 128 64 — — Milestones M1 (March 2026): 8-GPU baseline — DDP, profiling, observability M2 (May 2026): FSDP implementation, 16-GPU scaling tests M3 (July 2026): 128-GPU runs, fault tolerance, cost analysis M4 (August 2026): Full benchmark suite, architecture writeup, public repo Related Posts Posts will appear here as they\u0026rsquo;re published.\nGitHub: Available August 2026\n","permalink":"https://zhanyl-tech.github.io/projects/distributed-training-platform/","summary":"Building production-grade distributed training infrastructure across 128 GPUs — orchestration, fault tolerance, and cost optimization for large model training.","title":"128-GPU Distributed Training Platform"},{"content":"github.com/Zhanyl-tech/ib-slurm-exporter · Go · MIT\nmake demo No InfiniBand, no cluster, no Linux required — it builds a synthetic /sys, /proc and cgroup tree and serves real metrics off it.\nThe problem A 64-node run drops from 4,200 to 900 samples/sec. GPUs look busy. Slurm says RUNNING. Your Slurm exporter reports job count, state, and runtime — none of which move.\nThe cause is one HCA retransmitting: packet_seq_err climbing on mlx5_1, stalling the all-reduce and idling every GPU behind it.\nFabric monitoring can see that counter. It just cannot tell you which job owns it, because nothing connects a Slurm job ID to a network device.\nThe join Neither Slurm nor the HCA knows about the other. The bridge turns out to be a file descriptor — any process doing RDMA holds one open on a uverbs device:\njob_918001/step_0/cgroup.procs → PID 41001 → /proc/41001/fd/3 → /dev/infiniband/uverbs0 → /sys/class/infiniband_verbs/uverbs0/ibdev → \u0026#34;mlx5_0\u0026#34; → /sys/class/infiniband/mlx5_0/ports/1/hw_counters/packet_seq_err That last hop goes through sysfs on purpose. uverbs3 does not have to mean mlx5_3 — the numbering is independent. A host where they happen to match is exactly the host where the shortcut looks correct and is wrong somewhere else.\nWhat it refuses to do InfiniBand counters are per-device. The HCA counts packets; it has no idea which process sent them.\nWhen two jobs share a node and both use mlx5_1, splitting port_xmit_data between them is not hard — it\u0026rsquo;s impossible. The information isn\u0026rsquo;t in the hardware.\nExporters that paper over this emit job-labelled series that are wrong in the worst way: a noisy neighbour\u0026rsquo;s retries land on a healthy job, so the metric misleads you precisely when you\u0026rsquo;re using it to debug an incident.\nSo there are two families. ib_slurm_job_* appears only when a job is the device\u0026rsquo;s sole user. ib_slurm_device_* always appears and is never job-labelled. A third series, ib_slurm_device_jobs, shows why attribution is missing instead of leaving a mysterious gap.\nSuppressing attribution never loses data. In the demo mlx5_1 is shared, and its 48,221 packet_seq_err are still exported — at device level, where they\u0026rsquo;re true.\nSlurm 26.05 broke the usual approach From the release notes:\ncgroup/v2 directory structures are now keyed off of SLUID and not the JobId.\nAnything doing Atoi on the segment after job_ now gets a parse error, or worse, a number that isn\u0026rsquo;t a job ID. Nothing errors at runtime. Series just stop appearing, or appear against the wrong job.\nAll three layouts are handled — v1, v2, and 26.05\u0026rsquo;s SLUID form. Identifiers are treated as opaque; non-numeric ones resolve through scontrol, and anything that fails to resolve is counted, never guessed. A metric on the wrong job is worse than a metric on no job.\nI found this while writing up the 25.11 vs 26.05 comparison, which is the only reason this tool handles it — the approach I\u0026rsquo;d sketched a day earlier was already obsolete.\nWorth knowing separately: PIDs live in the step cgroups, not the job-level one. An exporter reading only job_*/cgroup.procs finds nothing and reports an empty cluster.\nHonest caveat The SLUID path is implemented from the release notes and has not been validated against a live 26.05 cluster. If you run one, I\u0026rsquo;d like to know what the real directory names look like.\nDesign note Every root — sysfs, proc, verbs, cgroup — is injectable. That\u0026rsquo;s what lets a Linux-only exporter be tested and demoed on a laptop, and it\u0026rsquo;s why the 14 tests run anywhere. It also means the synthetic tree is deliberately awkward: two jobs share an HCA so the suppression path is exercised, and one allocation is keyed by SLUID so the 26.05 layout is too.\n","permalink":"https://zhanyl-tech.github.io/projects/ib-slurm-exporter/","summary":"Correlate InfiniBand and RoCE counters with the Slurm job that owns them, so a slow collective can be traced to the fabric.","title":"IB Slurm Exporter"},{"content":"Status: 🔵 Design Phase — Deadline November 2026\nStack: CUDA · C++ · Python · QuantLib · NumPy · SciPy · Numba\nWhat This Is GPU-accelerated calibration of volatility surfaces for real-time derivatives pricing. Specifically: taking the computationally expensive problem of fitting local and stochastic volatility models to market data, and making it fast enough to run continuously as markets move.\nThis project sits at the intersection of my two domains — HPC infrastructure and quantitative finance — and is the bridge between them.\nThe Problem Volatility surface calibration is a core operation in options trading desks. You need to:\nTake observed market option prices across strikes and maturities Fit a model (Dupire local vol, Heston stochastic vol, or SABR) to these prices Use the fitted surface to price exotic options or hedge positions Do this continuously as market prices update (ideally sub-second) The calibration step — fitting the model — is where standard CPU implementations fall short. A full SPX volatility surface calibration can take 30-60 seconds on a single CPU core. That\u0026rsquo;s fine for end-of-day batch processing. It\u0026rsquo;s not fine for real-time trading.\nGoal: \u0026lt;1 second for full surface recalibration using GPU parallelism.\nWhy GPU Acceleration Works Here Volatility surface calibration involves:\nSolving PDEs across a grid of (strike, maturity) points — embarrassingly parallel Monte Carlo simulation for exotic option pricing — thousands of independent paths Nonlinear optimization with many function evaluations — each evaluation is a grid computation All three are naturally parallel. The CPU bottleneck is serialization, not compute. Moving to GPU eliminates that bottleneck.\nPlanned Architecture ┌──────────────────────────────────────────────────────┐ │ Market Data Input │ │ SPX option chain · Bid/Ask · Strikes · Maturities │ └──────────────────────┬───────────────────────────────┘ │ ┌──────────────────────▼───────────────────────────────┐ │ Implied Vol Extraction (CPU) │ │ Black-Scholes inversion · Interpolation │ └──────────────────────┬───────────────────────────────┘ │ ┌──────────────────────▼───────────────────────────────┐ │ GPU Calibration Engine (CUDA) │ ├──────────────────┬───────────────────────────────────┤ │ Dupire Local │ Heston Stochastic │ SABR │ │ Vol (PDE FDM) │ Vol (MC paths) │ (Analytic) │ │ │ │ │ │ CUDA kernel: │ CUDA kernel: │ Vectorized │ │ parallel grid │ parallel path │ CPU+GPU │ │ evaluation │ simulation │ │ └──────────────────┴──────────────────────┴────────────┘ │ ┌──────────────────────▼───────────────────────────────┐ │ Calibrated Surface Output │ │ Local Vol σ(K,T) · Stochastic Vol params │ │ Pricing engine ready · Greeks computation │ └──────────────────────────────────────────────────────┘ Technical Approach Local Volatility (Dupire) Dupire\u0026rsquo;s formula gives the local volatility surface directly from the implied volatility surface via a PDE. The computation is a grid evaluation — each point on the (K,T) grid is independent. CUDA maps naturally: one thread per grid point.\nHeston Stochastic Volatility (Monte Carlo) Heston requires Monte Carlo simulation for pricing: simulate thousands of paths of the underlying and variance process, compute payoffs, average. Classic GPU workload — each path is independent. Expecting 100-1000x speedup over single-core CPU.\nOptimization Loop (Levenberg-Marquardt) Both models require nonlinear optimization to fit model parameters to market prices. Each objective function evaluation is a GPU kernel call. The optimization loop runs on CPU, but the expensive inner loop runs on GPU.\nPlanned Benchmarks Model Operation CPU (single core) GPU (A100) Speedup Local Vol (Dupire) Surface calibration ~30s Target: \u0026lt;1s 30x+ Heston 10K path MC ~120s Target: \u0026lt;2s 60x+ SABR Surface fit ~5s Target: \u0026lt;0.2s 25x+ Connection to CQF Work This builds directly on my CQF projects:\nLocal vol calibration (Project 1) — production-grade this implementation Risk management (Project 3) — GPU-accelerated Monte Carlo for VaR The mathematical foundations are from CQF; the HPC implementation is the new contribution Milestones M1 (July 2026): CPU baseline — clean Python implementation with QuantLib validation M2 (September 2026): CUDA local vol kernel — first GPU implementation M3 (October 2026): Heston Monte Carlo on GPU, benchmark vs CPU M4 (November 2026): Full pipeline, benchmarks, public repo + writeup Related Posts Posts will appear here as they\u0026rsquo;re published.\nGitHub: Available November 2026\n","permalink":"https://zhanyl-tech.github.io/projects/gpu-volatility-surface/","summary":"Applying HPC and GPU computing to quantitative finance: calibrating volatility surfaces in real-time using CUDA kernels, finite difference methods, and Monte Carlo simulation.","title":"GPU-Accelerated Volatility Surface Calibration"},{"content":"github.com/Zhanyl-tech/gpu-reaper · Go · MIT\nmake demo No cluster, no GPUs, no risk — fake squeue, simulated telemetry, observe mode. Two jobs get classified as hung and escalate from alert to drain in about thirty seconds.\nThe problem A researcher requests 16 H100s for 48 hours. Four hours in, the training script deadlocks on an all-reduce because one rank died. The allocation is still held. Slurm is content: the job is RUNNING, the nodes are busy, the queue is moving.\nSlurm does not look at GPUs. It looks at allocations. That gap is 16 GPUs × 44 hours — 704 GPU-hours of a contended resource, consumed by a process doing nothing.\nWhy it will not kill your job The whole design follows from an asymmetry: killing a healthy job is far worse than letting a wasted one run another hour. A researcher whose 40-hour run dies at hour 39 loses the run and their trust in the tool.\nSo: observe mode is the default, enforcement is opt-in, and escalation requires sustained evidence. Concretely —\nJobs younger than the warmup are never judged. Container pulls and dataset staging legitimately show zero utilization. A breach must hold across the entire window. One busy sample clears the finding, because evaluation is on peak, not mean. A hole in the samples is treated as a collector fault, not idleness. This is the guard that stops a monitoring outage from cancelling the whole cluster. Cancel requires a previous cycle to have alerted and drained. No config change can jump a job straight to termination. Signatures Low utilization has several causes, and they deserve different responses.\nSignature Evidence Meaning Ceiling idle No memory, no processes, idle power Allocation is empty cancel hung Memory held, processes resident, zero compute Deadlocked collective cancel starved Memory held, low but nonzero compute Dataloader bottleneck alert unknown Breaching, no match Not modelled alert Starvation caps at alert deliberately. A slow dataloader is a tuning problem, and cancelling someone\u0026rsquo;s run over one is how a tool gets uninstalled.\nWhat building it taught me Two bugs worth recording, both found by CI rather than by me:\n.gitignore matches at any depth. A bare bin/ silently excluded demo/bin/, so the fake squeue the demo depends on never left my laptop. The demo the README promised could not have worked for anyone who cloned it. The fix is anchoring — /bin/.\n$(PWD) is not guaranteed in make. It comes from the environment; on the CI runner it expanded to empty and the demo\u0026rsquo;s PATH prepend silently did nothing. $(CURDIR) is make-native and always set.\nNeither is exotic. Both were invisible locally and obvious the moment something other than my machine ran the code — which is the argument for having the demo itself be a CI job.\nLimitations GPU samples identify a node, not a job. Where a node hosts several GPU jobs, attribution is ambiguous and the reaper skips them rather than guess — a false alert on a healthy job costs more than a missed finding costs GPU-hours. Correct attribution needs the Slurm cgroup hierarchy to map GPU PIDs back to job IDs, which is the subject of the next tool.\nIt reads nvidia-smi rather than DCGM, so utilization is coarse: a kernel occupying one SM reports the same 100% as a saturated device. That is exactly why high utilization is never treated as proof of health — only low utilization as evidence of a problem.\n","permalink":"https://zhanyl-tech.github.io/projects/gpu-reaper/","summary":"Find wasted GPU allocations on a Slurm cluster and escalate them through alert, drain, and cancel.","title":"GPU Reaper"},{"content":"github.com/Zhanyl-tech/slurm-scheduler-lab · MIT · Python 3.10+\nChanging PriorityWeightFairshare on a production cluster is a slow, expensive experiment: the feedback loop is days long, the blast radius is everyone\u0026rsquo;s queue, and the only rollback signal is people complaining. This runs the same policy against a workload in about a second.\nWhat it models Multifactor priority — the real formula from priority/multifactor, with age, fairshare, job size, partition, and QOS factors each normalised to [0,1] so the weights alone decide what the queue optimises for. Fairshare uses Slurm\u0026rsquo;s classic F = 2^(-U/S) with exponential usage decay.\nEASY backfill — priority-ordered dispatch with shadow-time reservations. The scheduler is never allowed to read a job\u0026rsquo;s true runtime, only its requested time_limit, which is what makes over-requesting visible as a measurable cost.\nUsing it on a real cluster sacct -a -X --parsable2 --starttime=now-30days \\ --format=JobID,Account,Submit,Elapsed,Timelimit,NNodes,ReqCPUS,ReqTRES \\ \u0026gt; trace.txt schedlab --sacct trace.txt --slurm-conf /etc/slurm/slurm.conf --nodes 64 It reads PriorityWeight* straight out of the config you are about to deploy, so you can compare the config you have against the config you are considering.\nWhat it found backfill off backfill on CPU utilization 72.2 % 83.6 % Mean wait 1913.0 min 373.7 min Median wait 2303.3 min 131.9 min Bounded slowdown 373.18 55.66 Sweeping PriorityWeightJobsize across 100,000x moves mean wait about 40%. Backfill moves it 400%. The number that actually explains the queue turned out to be time-limit accuracy — jobs using ~32% of the wall-clock they request.\nWritten up in Your Slurm priority weights matter less than your users\u0026rsquo; time limits.\nScope Deliberately not a Slurm reimplementation: first-fit node selection with no topology awareness, aggregate rather than per-node backfill reservations, flat fairshare, no preemption or gang scheduling. Good for comparing policies against each other on the same workload; not for predicting absolute wait times.\n23 tests cover reservation safety, fairshare maths, capacity invariants, and determinism across identical seeds.\n","permalink":"https://zhanyl-tech.github.io/projects/slurm-scheduler-lab/","summary":"Test Slurm priority and backfill policy against a job trace before it reaches a live controller.","title":"Slurm Scheduler Lab"},{"content":"I had a plan. Move a 250-node cluster from 25.05 to 25.11, write it up, ship it.\nThen I checked the release list and found 26.05 had gone out on July 14, along with 25.11.7. Twelve days ago. And 26.05 upgrades directly from 25.05 — which means the version I was carefully planning to land on is not the only option, and possibly not the right one.\nSo this is the comparison I actually needed: three versions, what breaks between them, and which one a cluster with one head node and 250 compute nodes should be aiming at.\nThe short version 25.05 25.11 26.05 Status current on my cluster stable, .7 out latest, .2 out Direct upgrade from — 25.05, 24.11, 24.05 25.11, 25.05, 24.11 Config renames — JobContainerType ExclusiveUser, ExclusiveTopo Prometheus none basic openmetrics + GPU allocation stats cgroup layout JobId JobId SLUID REST v0.0.41 deprecated v0.0.41 → removed v0.0.42 deprecated Both 25.11 and 26.05 are reachable in one hop from 25.05. That is the whole decision: one change window or two.\nThe renames that fail on restart Two of them, one per version, and both are the same class of problem — your config parses fine today and does not parse after the daemon restarts.\n25.11 renamed JobContainerType to NamespaceType. The job_container plugin interface became namespace, with a new namespace/linux plugin that handles filesystem, PID, and user namespaces. If you do per-job private /tmp — and if you have users, you do — this line is in your config.\n26.05 collapsed ExclusiveUser and ExclusiveTopo into a single Exclusive=[NO|NODE|USER|TOPO].\nNeither is hard. Both are a grep:\ngrep -rnE \u0026#39;JobContainerType|ExclusiveUser|ExclusiveTopo\u0026#39; /etc/slurm/ The reason to care is when you find out. A config-parse failure surfaces when slurmctld restarts, which on a planned upgrade is the exact moment you have a change window open, 250 nodes drained, and a room full of people waiting. Find it on a Tuesday instead.\nThe one that will break your tooling This is the change I would have missed, and it is the expensive one:\ncgroup/v2 directory structures are now keyed off of SLUID and not the JobId.\nRead that again if you own any monitoring that walks cgroups.\nThe standard way to attribute a process — or a GPU, or an InfiniBand counter — back to a Slurm job is to walk /sys/fs/cgroup/.../slurm/uid_*/job_*/ and match PIDs. Every job-aware exporter I know of does some version of this. In 26.05 that path is keyed by SLUID, not JobId, so anything parsing a job ID out of the directory name gets a value that is not a job ID.\nIt will not error. It will return the wrong number, or nothing, and your dashboards will go quietly flat.\nI have a direct stake in this one: I have been building a GPU-waste reaper whose next milestone is exactly this cgroup walk, and the note above means the implementation I sketched is already obsolete for anyone on 26.05. Better to learn that from a release note than from a silent regression six months in.\nThe API changed underneath it too — slurm_step_id_t replaces bare job_id in calls, so the API can be queried by SLUID. There is a SLURM_BACKWARD_COMPAT define if you compile against the C API, which is a kindness, but it is a deprecation runway rather than a permanent fix.\nThe good part 26.05 expands the openmetrics endpoints with GPU allocation statistics.\n25.11 added native Prometheus export from slurmctld — queue depth, job states, node states, scheduler cycle timing. Useful, and it kills the fragile shell-out-to-sinfo-and-parse sidecar that a lot of sites still run.\n26.05 adds GPU allocation to that. Which matters more than it sounds, because \u0026ldquo;how many GPUs are allocated\u0026rdquo; and \u0026ldquo;how many GPUs are doing work\u0026rdquo; are different questions and the gap between them is where cluster money goes. Native export answers the first for free. The second still needs someone reading NVML, but having the denominator without deploying anything is a real improvement.\nAlso in 26.05: srun --async, which queues step processes through stepmgr instead of requiring you to keep hundreds of backgrounded srun processes alive. If you have ever watched a workflow tool fork a thousand sruns and then watched the login node fall over, this is for you. Plus dynamic memory resizing — a running job can hand memory back via scontrol update, with sbatch --mem-update=\u0026lt;margin\u0026gt;@\u0026lt;delay\u0026gt; to automate it — and new topology/ring and topology/torus3d plugins.\n25.11\u0026rsquo;s headline was expedited requeue: --requeue=expedite puts a job that died to node failure straight back at the top of the queue, and holds its previous nodes so nothing else grabs them. For a training run that fell over at hour 40, that is the difference between restarting now and restarting behind everything that queued up while it ran.\nOne caution on it. Expedited requeue also triggers when the batch script exits non-zero and an Epilog fails. But a failing Epilog is frequently how you learn a node is sick. Wire \u0026ldquo;Epilog failed\u0026rdquo; to \u0026ldquo;requeue at top priority and hold those nodes\u0026rdquo; and a genuinely bad node can pin an expedited job in a loop. If you turn this on, your Epilog health check needs to drain decisively, not just exit non-zero.\nWhich one For this cluster, 26.05.\nThe upgrade path is the same length either way — one hop from 25.05 — and taking 25.11 means doing the whole exercise again in a few months to get to a release that is already out. The 250-node slurmd roll is the expensive part, and it costs the same whichever target you pick. Paying it twice for no reason is the easiest mistake to avoid here.\nThe argument against is real, though: 25.11 has had eight point releases and 26.05 has had two. If your cluster is the one people ship from and you cannot tolerate being an early adopter of a .2, take 25.11 now and 26.05 in six months. That is a legitimate risk call, not a cop-out — it just costs you two change windows.\nWhat I would not do is default to 25.11 because it was the plan before I looked.\nOrder of operations Nothing exotic, and unchanged across both versions:\ngrep the config for JobContainerType, ExclusiveUser, ExclusiveTopo. Audit anything talking REST for a pinned API version. v0.0.41 is gone in 26.05; v0.0.42 is deprecated there and goes in 26.11. Audit anything walking cgroups. This is the new one, and it is the one that fails silently. Back up slurmdbd. The database migration is the step with no easy undo, and it is the step people skip because it has always worked before. slurmdbd → verify → slurmctld → verify. Roll slurmd in batches, draining ahead of each. Never a compute node ahead of the controller. New features after the version move is stable. Expedited requeue and the GPU metrics endpoints are not part of the upgrade; they are the next change. I am writing that up properly — node batching, rollback triggers, the actual runbook — as its own repo.\nWhat this is and isn\u0026rsquo;t This is release notes plus a decision, not a war story. I have not run 25.05 → 26.05 on 250 nodes yet. When I do, the useful post is the delta between this plan and what actually happened, because that gap is where the real content always is.\nIf you have already made either hop, I would like to know what bit you — particularly whether the cgroup change broke anything you did not expect.\nSources: Slurm 26.05 release notes · 26.05 RELEASE_NOTES.md · 25.11 RELEASE_NOTES.md · 26.05.2 / 25.11.7 announcement · Upgrade guide\n","permalink":"https://zhanyl-tech.github.io/experiments/2026-07-26-slurm-25-11-upgrade-notes/","summary":"Three versions, two config renames that break on restart, and one cgroup change that quietly breaks every job-attribution tool you own.","title":"I Was Planning a Slurm 25.11 Upgrade. Then 26.05 Shipped."},{"content":"I wanted to change PriorityWeightFairshare on a cluster and could not justify the experiment. The feedback loop is days long, the blast radius is everyone\u0026rsquo;s queue, and the only rollback signal is people complaining in Slack.\nSo I wrote the thing that lets you try it offline: slurm-scheduler-lab. It implements Slurm\u0026rsquo;s priority/multifactor formula and EASY backfill, reads PriorityWeight* straight out of a slurm.conf, and replays either a synthetic workload or a real trace from sacct.\nThen it kept telling me I was tuning the wrong thing.\nFirst, backfill is doing most of the work Baseline, 300 jobs on 16 nodes:\nbackfill OFF backfill ON cpu utilization 72.2 % cpu utilization 83.6 % mean wait 1913.0 min mean wait 373.7 min median wait 2303.3 min median wait 131.9 min bounded slowdown 373.18 bounded slowdown 55.66 An 11-point utilization swing and a 5x cut in mean wait. Median drops 17x, because backfill disproportionately rescues the small jobs that were stuck behind a wide one. None of this is new — it\u0026rsquo;s why EASY backfill won in the nineties — but it sets the scale for everything after it.\nThen the weights, which move much less Sweeping PriorityWeightJobsize across four orders of magnitude:\nweight=0 util 87.3% mean wait 352.9 min p95 1978.4 min slowdown 52.08 weight=1000 util 90.6% mean wait 348.2 min p95 1879.4 min slowdown 58.08 weight=10000 util 83.6% mean wait 373.7 min p95 1817.7 min slowdown 55.66 weight=100000 util 93.4% mean wait 496.7 min p95 1904.7 min slowdown 60.79 There is a real tradeoff here, and it is the one you would predict: at weight=100000 the machine is fullest (93.4%) and the queue is slowest (497 min mean wait). Big jobs go first, utilization looks great on the dashboard, and everything small waits behind them.\nBut look at the range. Across a 100,000x change in the weight, mean wait moves by about 40%. Backfill moved it by 400%.\nPriorityWeightFairshare was flatter still — it redistributes wait between accounts, which is its job, without changing the total much.\nThe number that actually explains the queue Both runs above report the same line:\ntime-limit accuracy 32.2 % Jobs use about a third of the wall-clock they ask for. That is a property of my generator, not a measurement — I set the padding to ~3x because that is the range published trace studies keep reporting. Whether it holds on your cluster is a one-line sacct query, and I\u0026rsquo;d rather you check than take my word for it.\nIt matters because backfill plans against what users request, not what their jobs need. The scheduler computes a shadow time — the earliest moment the blocked job can start, assuming every running job runs to its full limit — and then only backfills work it can prove will not delay that reservation.\nA job that will really take 50 seconds but claims 500 is unprovable. It sits.\nThat\u0026rsquo;s the case I pinned in a test, because it\u0026rsquo;s the whole argument:\ndef test_backfill_plans_against_time_limit_not_true_runtime(): honest = [..., job(3, 2, duration=50, nodes=1, time_limit=50)] padded = [..., job(3, 2, duration=50, nodes=1, time_limit=500)] assert honest[2].start_time == pytest.approx(2.0) assert padded[2].start_time \u0026gt; honest[2].start_time Identical real runtimes. Identical cluster. Identical priority weights. The only difference is what the user typed into --time, and one of them backfills instantly while the other waits for the reservation to clear.\nWhat I\u0026rsquo;d do differently I came in expecting to ship a weights change. What I\u0026rsquo;d actually do first:\nMeasure time-limit accuracy on the real trace. One sacct query. If it\u0026rsquo;s near 30%, that\u0026rsquo;s the finding. Fix the defaults before the weights. A partition DefaultTime that matches reality beats any weight I sweep here. Then tune weights — for who waits, which is what they genuinely control, not how long the queue is. The simulator is on GitHub. Point it at your own slurm.conf and sacct output:\nsacct -a -X --parsable2 --starttime=now-30days \\ --format=JobID,Account,Submit,Elapsed,Timelimit,NNodes,ReqCPUS,ReqTRES \\ \u0026gt; trace.txt schedlab --sacct trace.txt --slurm-conf /etc/slurm/slurm.conf --nodes 64 I\u0026rsquo;d genuinely like to know whether the 32% holds up on a production trace, or whether my synthetic workload is flattering itself.\nCaveats This is a model, not a controller. Node selection is first-fit with no topology awareness, backfill reservations are computed on aggregate CPU/GPU counts rather than per node, fairshare is flat rather than hierarchical, and there\u0026rsquo;s no preemption or gang scheduling. Those simplifications are listed in the README because they bound what the numbers above are allowed to claim: they\u0026rsquo;re good for comparing policies against each other on the same workload, and not for predicting absolute wait times on your cluster.\n","permalink":"https://zhanyl-tech.github.io/experiments/2026-07-26-slurm-backfill-time-limits/","summary":"I built a simulator to tune Slurm priority weights. It kept telling me the weights were not the problem.","title":"Your Slurm Priority Weights Matter Less Than Your Users' Time Limits"},{"content":"Sparse attention is easy to talk about and hard to ship. The question that actually matters isn\u0026rsquo;t \u0026ldquo;is attention sparse?\u0026rdquo; — it\u0026rsquo;s which structure you can exploit cheaply enough that the win survives contact with a GPU kernel.\nTwo papers attack this from opposite directions:\nLServe starts from the serving system and asks: what sparse mechanisms can I plug in at prefill and at decode that give real throughput? SampleAttention starts from the attention map and asks: what sparse patterns actually show up in real models, and how do I select them without running dense attention first? The figures below follow the usual paper-style schematic: minimal decoration, explicit labels, and placement next to the claims they support (cf. NeurIPS / ICML figure guidelines: vector line art, legible fonts at print size, one idea per panel).\nRead together they fit: SampleAttention gives you the map of where sparsity lives; LServe gives you the system that exploits it.\nTL;DR LServe: different sparse mechanisms for different bottlenecks. Prefill skips KV blocks. Decode prunes KV pages. SampleAttention: two patterns explain most of the important attention mass during prefill. Columns = global anchors. Slashes = local continuity. CRA: a runtime accuracy guardrail. Measures how much attention mass your sparse mask actually keeps. No offline profiling required. Why Fine-Grained Sparsity Doesn\u0026rsquo;t Work Both papers start by rejecting the same naive idea: drop arbitrary individual scores below some threshold, get a faster kernel.\nThe problem is GPU execution. Attention kernels iterate sequentially over KV blocks. Skipping a few scalar operations inside a block doesn\u0026rsquo;t save iterations — the block still has to be loaded. Unstructured sparsity looks good on FLOPs and bad on actual wall-clock time.\nBoth papers land on the same fix: exploit structure that aligns with how GPUs already process attention.\nLServe — Prefill Phase The prefill bottleneck is sequential KV iterations. You have a long prompt, many query tokens, and each needs to walk the KV history. The speedup has to come from reducing how many KV blocks each query touches, not from finer-grained skipping within blocks.\nLServe\u0026rsquo;s approach: block-sparse attention where each block is either fully kept or fully skipped. No partial blocks. This is an important design call — it\u0026rsquo;s exactly the granularity that maps to the kernel\u0026rsquo;s iteration structure.\nOn top of that, LServe converts roughly half the heads into streaming heads with a Λ-mask. A streaming head only keeps recent tokens plus a small set of sink tokens. It never touches the bulk of the KV history. These heads are almost free — they skip the expensive middle of the attention computation entirely.\nSo prefill is doing two things simultaneously:\nBlock-level skipping on the heads that still run full attention. Lambda-mask on half the heads so they don\u0026rsquo;t run full attention at all. Both still go through unified kernels, not separate code paths.\nFigure 1. LServe prefill: (a) block-sparse attention — entire KV blocks are either computed or skipped at kernel tile granularity; (b) streaming head with a Λ-shaped support (sink tokens plus a local window, middle of the sequence not materialized). LServe — Decode Phase Decode is a different problem. Each step is one query token attending to the full KV cache. There\u0026rsquo;s no quadratic cost, but there\u0026rsquo;s a serious bandwidth cost: loading the KV cache from HBM every single step. At 128K context, that\u0026rsquo;s a lot of data movement per generated token.\nLServe\u0026rsquo;s claim is that for any given query token, you don\u0026rsquo;t need the full KV cache. Only a small number of pages are actually relevant, and that count stays roughly constant even as context grows. So instead of loading everything, you select the important pages.\nThe mechanism: a hierarchical page selector scores all pages cheaply using a summary vector per page, picks the top-k, then does fine-grained attention only within those. Selection results are reused across a small window of decode steps to avoid paying the selection cost on every single token.\nCombined with INT4 KV quantization, the gains stack: sparsity cuts the number of pages you touch; quantization cuts the byte cost of each page you do touch.\nPrefill and decode are genuinely different optimization problems. LServe treats them that way.\nFigure 2. Decode: coarse per-page scoring, top-k subset, then token-level attention only inside selected pages. Constant-ish k vs sequence length is the intended scaling behavior; INT4 KV further reduces bytes per read. SampleAttention — What Sparse Patterns Are Real SampleAttention focuses on prefill, specifically the TTFT (time to first token) problem at long context. Its core empirical point: attention patterns vary across heads, layers, and prompts, but two structures dominate in almost every setting.\nColumn Pattern Certain key positions attract high attention from many different query positions — a whole column of the attention matrix lights up. These are the attention sinks: globally important tokens (separators, special tokens, early context) that many queries want to keep regardless of what they\u0026rsquo;re looking for.\nIf you\u0026rsquo;re going to keep any sparse structure, keep the hot columns. That\u0026rsquo;s where the long-range global information lives.\nSlash Pattern Query tokens tend to attend to keys near them in the sequence — diagonal or near-diagonal structure. This is local context: the token next to yours matters more than the token 10K positions back. In the attention matrix, that shows up as a slash-shaped band tracking the diagonal.\nColumns + slashes together cover a surprisingly large fraction of the attention mass in real models. The paper doesn\u0026rsquo;t need a long catalog of special cases — just these two.\nFigure 3. Schematic causal attention map: column-heavy keys (global anchors) and mass concentrated near the diagonal (local continuity). The actual heatmaps are head- and layer-dependent; the figure isolates the two recurring structures discussed in the paper. CRA — The Accuracy Proxy Most sparse attention papers tune for a fixed sparsity ratio. SampleAttention uses a different control knob: Cumulative Residual Attention (CRA).\nCRA measures, for each query, how much of the original attention probability mass remains after applying the sparse mask. Then it takes the minimum across all queries.\nThat last part matters. It\u0026rsquo;s not asking whether the average query kept enough. It\u0026rsquo;s asking whether the worst-case query kept enough. One badly damaged query can degrade a generation even if 99% of queries are fine.\nThe practical benefit: CRA lets you set a quality floor instead of a sparsity budget. You say \u0026ldquo;no query should lose more than 10% of its attention mass\u0026rdquo; rather than \u0026ldquo;always keep 20% of tokens.\u0026rdquo; On hard prompts where attention is concentrated, the mask expands. On easy prompts, it shrinks. You get adaptivity for free.\nHow it\u0026rsquo;s computed without dense attention: SampleAttention samples a few query blocks spread across the sequence. From those sampled queries it accumulates approximate column and slash scores cheaply. It then picks the minimum column/slash budget that hits the CRA threshold, and merges those into the final mask. No offline profiling needed; it adapts per-head, per-layer, per-prompt at runtime.\nFigure 4. CRA aggregates per-query retained softmax mass under the sparse mask and takes a minimum across queries (worst-query control). The method compares a threshold τ to that aggregate rather than fixing a static token count; see the paper for the exact estimator and sampling schedule. How They Fit Together SampleAttention answers the structure question: here are the two patterns that matter, here\u0026rsquo;s how to identify them cheaply, here\u0026rsquo;s the accuracy signal that tells you if your mask is good enough.\nLServe answers the systems question: now that structured sparsity is real, here\u0026rsquo;s how you build a serving stack around it — unified kernels that handle both sparse and streaming heads, a KV page-selection mechanism for decode, and KV quantization to amplify the bandwidth gains.\nThe pairing matters because most sparse attention work stops at prefill TTFT. But production serving can\u0026rsquo;t stop there. Long outputs shift the bottleneck to decode — and decode needs a completely different sparse mechanism than prefill.\nWhy It Matters in Production If you\u0026rsquo;re building long-context inference infrastructure, the real lesson is that the bottleneck moves over the lifetime of a request.\nAt prefill: you\u0026rsquo;re walking a long KV history over many query tokens. Sparsity needs to cut KV block iterations. At decode: you\u0026rsquo;re repeatedly touching a huge KV cache for one token at a time. Sparsity needs to cut page reads. Treating both with the same mechanism leaves gains on the table.\nThree Takeaways Prefill and decode need different sparse mechanisms. They have different bottlenecks; don\u0026rsquo;t optimize them the same way. The useful sparse patterns are column and slash. Column = global anchors. Slash = local continuity. Together they cover most of the mass. Use CRA as your accuracy guardrail, not a fixed sparsity budget. It\u0026rsquo;s adaptive, it\u0026rsquo;s a worst-case bound, and it runs at inference time. References LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention: arXiv:2502.14866 SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention: arXiv:2406.15486 Related: How KV-Cache Paging Works in vLLM — and Why It Matters for Production\n","permalink":"https://zhanyl-tech.github.io/blog/2026-01-30-lserve-sampleattention-sparse-attention/","summary":"How LServe and SampleAttention use structured sparsity differently for prefill vs decode, and why CRA is the accuracy proxy you actually want.","title":"LServe and SampleAttention: What Sparse Attention Actually Changes in Prefill and Decode"},{"content":"Running inference benchmarks as part of the Agentic ML Inference Infrastructure project. This is a first-pass comparison — not a rigorous study, just enough to establish a baseline and understand what to optimize next.\nSetup Hardware: Single NVIDIA A100 80GB SXM Model: Llama-3 70B (BF16) Framework versions: vLLM 0.4.1, TensorRT-LLM 0.9.0 Workload: 512 input tokens → 128 output tokens, batch sizes 1 / 4 / 8 / 16 No quantization, no speculative decoding — baseline comparison first.\nRaw Numbers Batch Size vLLM (tok/s) TensorRT-LLM (tok/s) Delta 1 412 487 +18% 4 1,380 1,710 +24% 8 2,240 2,890 +29% 16 3,180 4,210 +32% TensorRT-LLM is consistently faster, with the gap widening at higher batch sizes. Expected — TensorRT\u0026rsquo;s AOT compilation and fused kernels are doing real work here.\nWhat\u0026rsquo;s Actually Happening vLLM\u0026rsquo;s PagedAttention is doing smart memory management (KV-cache paging), which is why it can handle more concurrent sessions before OOM. TensorRT-LLM trades that flexibility for raw throughput via compiled CUDA kernels.\nFor my use case (agentic infrastructure with many concurrent sessions), the vLLM memory management model may actually be worth the throughput hit. A 30% throughput reduction is recoverable. Running out of memory when session 87 spins up is not.\nWhat\u0026rsquo;s Missing From This Benchmark Continuous batching behavior: How does each handle the arrival pattern of real agentic workloads? Not uniform batch sizes. KV-cache pressure: At what session count does each start degrading? Latency percentiles: Throughput numbers hide p99 latency variance, which matters for tool call round-trips. Quantized comparison: INT8 and FP8 on TensorRT-LLM likely closes the gap with vLLM significantly. Next Steps Running the continuous batching test next week. That\u0026rsquo;s the number that actually matters for the agentic use case.\nPart of the Agentic ML Inference Infrastructure project.\n","permalink":"https://zhanyl-tech.github.io/experiments/2026-01-20-vllm-vs-tensorrt-llm-first-look/","summary":"Quick benchmark comparing vLLM and TensorRT-LLM throughput on a single A100 80GB with Llama-3 70B.","title":"vLLM vs TensorRT-LLM: First Throughput Numbers on Llama-3 70B"},{"content":"Most articles about vLLM lead with the benchmark numbers. I want to start with the problem it actually solves, because understanding why KV-cache paging exists is what makes the implementation decisions make sense.\nThe Problem: Memory Fragmentation Kills Concurrency When you serve an LLM in production, you\u0026rsquo;re not running one request at a time. You have dozens to hundreds of concurrent sessions, each at a different point in its generation. Each session needs to store its KV-cache — the key-value tensors computed from every token processed so far.\nThe naive approach allocates a contiguous block of GPU memory for each session\u0026rsquo;s KV-cache at the start of the request. This causes two problems.\nProblem 1: You have to pre-allocate for the worst case. If your model supports 8K context and you pre-allocate for the maximum, most sessions use a fraction of that. A 100-token conversation is holding memory reserved for 8K tokens. GPU memory utilization sits at 20-30% even when you think the server is busy.\nProblem 2: Fragmentation blocks new sessions. After serving a few hundred requests, your GPU memory looks like Swiss cheese — many small free blocks that collectively add up to enough space for a new KV-cache, but no single contiguous block large enough to fit it. New requests queue up even though memory is technically available.\nThis is exactly the same problem that motivated virtual memory and paging in operating systems. vLLM\u0026rsquo;s PagedAttention applies the same solution.\nPagedAttention: Paging for KV-Caches PagedAttention divides GPU memory into fixed-size physical blocks. Each block holds KV tensors for a fixed number of tokens — typically 16. A session\u0026rsquo;s KV-cache is stored in a linked sequence of physical blocks, which don\u0026rsquo;t need to be contiguous.\nA block table maps each session\u0026rsquo;s logical KV positions to physical block addresses. The attention computation is modified to follow this indirection — instead of reading from a contiguous buffer, it reads from wherever the block table points.\nSession A KV-cache: Logical block 0 → Physical block 47 Logical block 1 → Physical block 12 Logical block 2 → Physical block 91 Session B KV-cache: Logical block 0 → Physical block 3 Logical block 1 → Physical block 58 The blocks themselves are small and fixed-size, so they can be allocated from a pool. When a session completes, its blocks are returned to the pool and become immediately available to new sessions. No fragmentation.\nWhat This Changes in Practice The practical effect is significant. Memory utilization jumps from ~30% with naive allocation to ~90%+ with PagedAttention, on the same GPU, with the same model.\nFor the same hardware, that means you can serve 3x more concurrent sessions. Or alternatively, you can run a larger model and still hit your concurrency target.\nThere\u0026rsquo;s another benefit that matters for agentic workloads specifically: prefix caching becomes practical. When multiple sessions share the same system prompt (common in multi-agent deployments), those prefix blocks can be shared across sessions — physically shared, not copied. Sessions that differ only in the user-turn portion of their context can share all the prefix KV-cache blocks.\nThe Attention Kernel Change The attention computation itself needs modification to work with paged KV-caches. Standard FlashAttention reads from a single contiguous KV tensor. PagedAttention\u0026rsquo;s kernel takes the block table as an additional input and gathers blocks during the attention computation.\nThis is where the CUDA work lives. The kernel processes each query position by looking up which physical block contains its corresponding key-value data, loading that block, and performing the attention dot products. The scatter/gather pattern adds some overhead compared to reading from contiguous memory, but it\u0026rsquo;s small relative to the memory savings.\nWhen This Matters Most Not every deployment benefits equally from PagedAttention. Short sessions with simple prompts on high-throughput pipelines may see minimal benefit. The gains are largest when:\nSessions have widely varying lengths (long-tail distribution) Many sessions share the same system prompt or prefix You\u0026rsquo;re running close to GPU memory capacity with concurrent requests Generation length is unpredictable at request time That last point is particularly relevant for agentic use cases, where reasoning chains can expand significantly as the agent encounters unexpected tool results. Pre-allocating for a worst-case chain length isn\u0026rsquo;t practical when you don\u0026rsquo;t know what the chain looks like.\nWhat I\u0026rsquo;d Look for Next The benchmark implications of all this are non-obvious. Throughput numbers that measure tokens-per-second on a single batch don\u0026rsquo;t capture the memory utilization improvements. The right benchmark for a production serving system is: how many concurrent sessions at what latency percentile, at what memory utilization?\nThat\u0026rsquo;s the test I\u0026rsquo;m running next, comparing vLLM against TensorRT-LLM under real agentic traffic patterns rather than synthetic batch benchmarks.\nRelated: vLLM vs TensorRT-LLM: First Throughput Numbers — the benchmark companion to this post.\nPart of the Agentic ML Inference Infrastructure project.\n","permalink":"https://zhanyl-tech.github.io/blog/2026-01-15-kv-cache-paging-vllm/","summary":"A technical walkthrough of PagedAttention: the memory management innovation that makes vLLM practical for serving many concurrent LLM sessions without fragmenting GPU memory.","title":"How KV-Cache Paging Works in vLLM — and Why It Matters for Production"},{"content":"Who I Am I\u0026rsquo;m a senior HPC and cluster infrastructure engineer at a quantitative trading firm with 8+ years building and operating large-scale distributed systems — Kubernetes, SLURM, GPU clusters, and cloud platforms.\nI\u0026rsquo;m pursuing an MS in Computer Science with an ML specialization at Georgia Tech, alongside the Certificate in Quantitative Finance (CQF). This gives me production infrastructure expertise, ML systems depth, and the mathematical foundations for derivatives pricing and risk modeling.\nWhat I Build I work on the infrastructure that makes ML systems run fast and reliably at scale: GPU cluster operations, inference optimization (vLLM, TensorRT-LLM), distributed training orchestration, and the emerging agentic AI infrastructure layer — deploying autonomous systems that reason, decide, and act on production workloads.\nWhat I Write About This site has two main content streams:\nBlog — Technical deep dives on ML infrastructure, GPU inference, distributed training, and the systems engineering behind quantitative platforms. Written from a practitioner\u0026rsquo;s perspective.\nLab Notes — My public lab notebook. Shorter field reports on deploying new tools, running benchmarks, and documenting what I find before anyone else does.\nI also maintain project pages for my portfolio work, notes for quick technical observations, and a reading list of books I recommend.\nCurrent Focus Production: Expanding ML infrastructure scope — vLLM inference optimization, GPU cluster operations Academic: MS in CS (ML specialization, Georgia Tech, May 2027) + CQF final project (July 2026) Portfolio: Three projects at the intersection of HPC inference, distributed training, and quantitative finance Learning: Agentic AI infrastructure (LangGraph, MCP, NemoClaw/OpenClaw), hardware-agnostic inference (vLLM, Ray, JAX) Outside of engineering, I play chess and poker — both sharpen the strategic and probabilistic thinking I apply to systems architecture and risk modeling.\nResume 📄 Download Resume (PDF)\nContact GitHub LinkedIn X / Twitter Resume: Download PDF ","permalink":"https://zhanyl-tech.github.io/about/","summary":"About me","title":"About"},{"content":"Books that shaped how I think about systems, markets, and engineering. Updated quarterly.\nSystems \u0026amp; Engineering High Output Management — Andy Grove Treating organizations like engineering pipelines. The management book that reads like a systems design doc. Essential for anyone who manages infrastructure or people.\nDesigning Machine Learning Systems — Chip Huyen The production ML bible. If you\u0026rsquo;re deploying models, not just training them, start here.\nAI Engineering — Chip Huyen The 2025 follow-up. Covers the full stack from inference to deployment in the era of foundation models.\nQuantitative Finance The Man Who Solved the Market — Gregory Zuckerman How Jim Simons and Renaissance Technologies proved that elite scientists and HPC engineers could systematically extract alpha from financial markets. The history of what you\u0026rsquo;re building toward.\nDecision-Making \u0026amp; Strategy Thinking in Bets — Annie Duke Professional poker player on decision-making under uncertainty. Directly applicable to risk management, trading decisions, and engineering tradeoffs.\nLast updated: March 2026. This list only includes books I\u0026rsquo;d recommend to a peer engineer working at the intersection of ML infrastructure and quantitative systems.\n","permalink":"https://zhanyl-tech.github.io/reading/","summary":"Books I recommend","title":"Reading"}]