vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes

Every performance number in this post was measured on our own two DGX Sparks and is labeled with the most precise identifiers we have for that run — image digest or pinned gitsha, checkpoint, and date. Where we cite someone else’s number, we say so. Where we were wrong earlier, the timeline says that too.

Why this post exists

The DGX Spark’s serving ecosystem moved faster than anyone’s documentation could — NVIDIA’s launch guidance and the vLLM team’s Spark deep-dive were each accurate for the moment they were written, and the platform kept moving underneath them. What that left for operators to discover in production: which image builds carry sm_121 kernels and which don’t, why some NVFP4 checkpoints crash or emit garbage on paths that work fine on datacenter Blackwell, what --gpu-memory-utilization really means on a unified 128 GB pool, and what happens to speculative decoding under concurrent load rather than in a batch-1 demo. We wrote our findings down after each incident. This post is that document, cleaned up, with the receipts — offered as a snapshot with dates attached, since it will age the same way.

Two boxes inform everything here: Spark-1 and Spark-2, both GB10 (aarch64, sm_121a, 128 GB unified, ~273 GB/s), serving production traffic for a homelab agent fleet — reasoning, tool-calling, and structured extraction, 24/7, behind a model gateway. This is not a benchmark rig; every lesson below came out of something breaking.

The five-minute model of what vLLM is doing on this box

Everything in this post follows from four pieces of vLLM machinery. If you already operate vLLM, skip ahead; if not, these four paragraphs are the vocabulary the rest of the post uses.

1. Serving has two phases with opposite bottlenecks. Prefill processes your whole prompt in parallel — it’s compute-bound, measured in thousands of tokens/second, and determines time-to-first-token. Decode generates one token at a time, and each token requires re-reading the model’s active weights from memory — it’s memory-bandwidth-bound, which on a 273 GB/s unified-memory box is the budget everything competes for. This split explains most of the numbers here: why a 35B MoE with only 3B active parameters decodes fast on a Spark (decode reads the active set, not the total), why a weight-only quantization like Marlin costs little at single-user decode but shows up at prefill and concurrency (those are compute-heavy), and why our tables always report the phases separately.

2. Kernels are chosen by a chain with three links — and any link can break. When vLLM loads a model it reads the checkpoint’s quantization config (config.json), runs it through a backend “oracle” that maps quant format + GPU architecture to a ranked list of candidate kernel backends — that’s the ['FLASHINFER_TRTLLM', … 'FLASHINFER_CUTLASS', … 'MARLIN', 'EMULATION'] line you’ll see in our boot logs — and then the winning backend still has to find a compiled kernel binary (cubin) for your GPU inside the wheel you actually installed. Three links: checkpoint declares, oracle selects, wheel delivers. Each fails differently: a W4A16 checkpoint means the oracle correctly selects Marlin (nothing to fix); a wheel without sm_121 cubins means the selected backend dies at runtime (no kernel image is available); and a version-gated integration mismatch can silently drop you onto slow fallbacks. Most of this post’s timeline is one of those three links breaking.

3. The KV cache is a pre-allocated pool, not a meter. vLLM (PagedAttention) grabs one big block of memory at startup, sized by --gpu-memory-utilization, and pages sequences into it. It’s capacity — the boot log tells you the pool size in GiB and in tokens, and dividing them gives the per-token cost that makes context-length planning arithmetic instead of folklore (worked example in the unified-memory section). On a unified-memory box the grab itself is the danger: it can’t be swapped, so an oversized pool is the OOM.

4. Speculative decoding buys speed with a coupled scheduler. A cheap drafter proposes k tokens; the full model verifies them in a single forward pass; accepted tokens cost almost nothing extra. Acceptance depends entirely on how predictable your output is to that drafter — code and structured JSON accept well, free prose doesn’t — and the metric vLLM reports (“acceptance length”) includes a guaranteed bonus token, so read it carefully. The part nobody advertises: the drafter is wired into the scheduler, so turning speculation on can change scheduling behavior itself — disable async scheduling, cap the token budget per step. A “free” flag can therefore tax prefill and TTFT even while decode acceptance looks healthy. That coupling is exactly how our n-gram experiment lost (full numbers below).

The timeline: from launch-day optimism to a config that survives

This is the part most guides skip: when each fact became true. sm_121 support moved so fast that advice written eight weeks apart contradicts itself — including our own. Dates below come from our git history, incident notes, and the vLLM release record.

2025-12-20 — the origin. NVIDIA’s launch-window vLLM container documentation (NGC vllm:25.11-py3 and its release notes) claimed clean “DGX Spark functional support” and NVFP4-on-Blackwell. We wrote our first Spark serving doc from those sources, taking the claims at face value. What wasn’t yet visible in that material: sm_121 is consumer Blackwell, not the sm_100 the FP4 kernel work had targeted, and none of the then-shipped FlashInfer wheels contained sm_121 cubins — details the launch-window docs preceded rather than concealed. Everything below is the gap between documentation written at launch and a working box, closing in real time.

2026-02 → 2026-05 — upstream catches up, in pieces. The vLLM release record tells the real support story: enable_sm120_family (#33517, Feb); “Fix DGX Spark logic” (#38126, → v0.19.0); disable the broken FlashInfer CUTLASS MoE on sm_121 as a stopgap (#39825, → v0.20.0); fix the wrong sm_121 exclusion from Marlin/CUTLASS FP8 (#35568) and land the b12x sm_121 kernels (#40082) — both only in v0.22.0 (2026-05-29). In our tested pre-v0.22 images and checkpoints, native NVFP4 paths crashed (cutlass FP4 gemm ... sm120/sm121 ... Error Internal), silently emitted garbage, or required Marlin — behavior varies with the image build, packaged wheels, checkpoint, and backend, so treat this as our observed floor, not a universal law.

2026-05-31 — the empirical break. After weeks of intermittent failures we isolated the pattern on our own boxes: prebuilt FlashInfer wheels shipped zero sm_120/121 cubins; the model-card env vars rerouted only MoE layers while dense NVFP4 linears still hit the broken CUTLASS path; and the reliable fix was forcing Marlin for both backends (plus --enforce-eager for hybrid-Mamba models, a graph-capture bug fixed later in v0.22.0). Forced-Marlin became our fleet default — correct at the time, and understandably carried forward by community guides — including ours — as if it were a permanent hardware rule.

2026-06-01/02 — reconciliation, then past Marlin. The vLLM team published its Spark deep-dive claiming auto backends were fine — which contradicted our validated config. Reconciling blog-vs-repo-vs-our-logs showed the disagreement was a version gap, not a dispute: our Marlin era ran on v0.20.0, which simply lacked the v0.22.0 kernels. On 2026-06-02 we moved to a digest-pinned cu130 nightly and --moe-backend auto selected FLASHINFER_CUTLASS native FP4 — coherent, no Marlin, no enforce-eager, and faster (~50.2 → ~55.8/58.5 tok/s on our Nemotrons). Meanwhile stock v0.22.0-cu129 — same vLLM version, different wheel — still crashed on auto because the cu129 wheel didn’t bake the sm_121 MoE cubin. That’s when “which vLLM version?” stopped being the right question and “does this wheel carry the cubins?” replaced it.

2026-06-05 → 2026-07-05 — the MTP production saga. We deployed the 35B Qwen with in-checkpoint MTP: 86.8% acceptance in validation, then CUDA error: an illegal memory access crashes every few hours in production — only under concurrent load (upstream #41190/#42084; the failure signature is a broken draft path poisoning the accepted-token buffer). An image bump to a 0.23-era nightly fixed the load-time bug (#35031 via #46316) but not the runtime crash, so spec-decode went off. Along the way, a second, unrelated checkpoint lesson: the W4A16 build’s 4-bit lm_head produced !!!! garbage on greedy structured-extraction output — quantization choices inside the checkpoint, not the engine, decided both failures.

2026-07-11/12 — the W4A4 resolution. Swapping to a genuine W4A4 checkpoint (4-bit activations, lm_head unquantized) moved experts and drafter onto the FLASHINFER_CUTLASS path — and the crash was gone: 8-way soak, 400/400, zero IMA, then re-enabled in production and bumped to num_speculative_tokens: 3 (mean acceptance length ~3.1 output tokens per verification step, bonus token included; ~95% draft acceptance on real traffic). The upstream bugs remain open; we didn’t patch vLLM, we routed around the crashing kernel path via configuration — checkpoint format and drafter backend together, so neither alone gets the credit. The transferable lesson stands regardless: on sm_121, the checkpoint’s quant format — W4A4 vs W4A16 — decides which kernels you run, and therefore which bugs you hit.

2026-07-19 — today. Both boxes on digest-pinned nightlies (0.22/0.23-era), native FP4, MTP where the model ships heads, everything below measured on this config. We smoke-tested the v0.25.1 stable candidates the same day this post was assembled — and the timeline’s lesson reproduced on cue: the official cu129 stable still ships no sm_121 cubins (dead at engine init), and the promising new cu13 tag turned out to be a mis-stamped dev artifact decoding 2.6× slow (details in the image table below). Seven months in, the rule hasn’t changed: there is still no viable aarch64 stable for GB10 native FP4; validate the wheel you actually pulled, then pin it.

The one rule that explains all the FP4 confusion

A widely-shared rule — reasonably derived from what its authors tested — says “GB10 has no native FP4 compute, so FP4 MoE must run on Marlin.” Our production logs show the rule has an important exception:

Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends:
  ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED',
   'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION']

That’s a Qwen3.6-35B-A3B NVFP4 MoE on --moe-backend auto, coherent output, running for days under real load, on a GB10. No Marlin anywhere.

Native FP4 on sm_121 needs two conditions, and both must hold:

  1. The image’s FlashInfer wheel actually bakes sm_121 cubins. Source support is not wheel support. Stock v0.22.0-cu129 has the sm_121 kernels in source (#40082) but the cu129 wheel ships without the MoE cubin — auto dies with no kernel image is available. The cu130 nightlies bake it. This is an image property; smoke-test the exact digest you deploy.
  2. The checkpoint is genuine W4A4 (nvfp4-pack-quantized: 4-bit weights and activations). A W4A16 weight-only “NVFP4” checkpoint has no FP4 activations, so there is no FP4 matmul to dispatch — auto lands on Marlin correctly, and no flag will change it.

Condition 2 is the one nobody talks about, and it explains the contradictory community reports. We ran three quantizations of the same model (Qwen3.6-35B-A3B) on the same image:

Checkpoint Format What auto picks
nvidia/Qwen3.6-35B-A3B-NVFP4 (modelopt) W4A16 weight-only MARLIN
RedHatAI NVFP4 build W4A4 FLASHINFER_CUTLASS
unsloth/Qwen3.6-35B-A3B-NVFP4 (deployed) W4A4 FLASHINFER_CUTLASS

So when a guide says Marlin is required, it’s worth checking which checkpoint was tested — most popular recipes use the W4A16 builds, which made the conclusion easy to reach. If it was an nvidia/... modelopt W4A16 build — most of the popular recipes are — their observation was correct for that checkpoint; it just doesn’t generalize into a hardware rule.

Decision tree:

Is the checkpoint W4A4 (nvfp4-pack-quantized, activations num_bits: 4)?
├─ NO (W4A16 weight-only) → auto → Marlin. This is correct; don't fight it.
│    At batch-1 you barely pay for it: decode is bandwidth-bound and 4-bit
│    weight storage already captures the win. Marlin costs you prefill + concurrency.
└─ YES → does your image's flashinfer wheel bake sm_121 cubins?
     ├─ NO (e.g. stock v0.22.0-cu129) → force --moe-backend marlin, or change image
     └─ YES (cu130 nightlies, pinned by gitsha) → --moe-backend auto
          → FLASHINFER_CUTLASS native FP4, dense AND MoE. Faster than Marlin,
            full cudagraphs, and (0.23-era) --attention-backend flashinfer
            works even on hybrid attention/Mamba/GDN models.

One more wrinkle from the release archaeology: the fastest sm_121 path, flashinfer_b12x (#40082), is intentionally excluded from auto-selection pending an upstream CUTLASS MMA-guard fix — so auto gives you a working-and-fast path, not necessarily the theoretical-fastest one. We haven’t needed b12x; CUTLASS is already ahead of Marlin.

Which images actually work (July 2026)

The single most common failure mode on Spark is an image whose wheels predate sm_121. Verified state as of 2026-07-19 (Docker Hub API + our deployments):

Image vLLM sm_121 verdict
vllm/vllm-openai:nightly-<gitsha> (cu130 line) 0.23.x-dev ✅ our Spark-2 production: native FP4 + MTP, pinned by gitsha
vllm/vllm-openai:nightly-4721bb3aa… 0.22.1rc1.dev26 ✅ our Spark-1 production (Nemotron ×2, native FP4)
vllm/vllm-openai:v0.25.1-aarch64-cu129-ubuntu2404 (stable, 2026-07-13) 0.25.1 ❌ tested 2026-07-19: dead at EngineCore init — CUDA error: no kernel image is available in the FlashInfer alloc path. cu129 stable wheels have now shipped without sm_121 cubins for three consecutive releases (0.22.0 → 0.25.1)
vllm/vllm-openai:vllm-arm64-cu13-0.25.1-<sha> (2026-07-17) says 0.25.1… ❌ tested 2026-07-19: a trap. Looks like the long-awaited cu13 aarch64 stable, and the sm_121 cubins ARE there (auto → FLASHINFER_CUTLASS, coherent output) — but /version reports 0.1.dev18498, a mis-stamped CI artifact, and decode ran 2.6× slower than our pinned 0.22-era nightly on the identical config (23.5 vs 62.0 tok/s, prefill halved too)
vllm/vllm-openai:cu130-nightly ~0.20-era ❌ dormant since 2026-04-23 — it still appears in guides written before the track went quiet (easy to miss; we only caught it via the Docker Hub API); the cu13 line moved to nightly-dev-arm64-cu13.0.1-<sha>
v0.22.0-aarch64-cu129 + forced Marlin 0.22.0 ✅ as a pinned stable fallback only (weight-only path, ~50 tok/s on our 30B Nemotron)

Rules that survived contact:

  • Check /version before anything else — a tag’s name is a claim, not a fact. The cu13 “0.25.1” tag above passed every kernel check and would have shipped a 2.6× decode regression; the one-line curl :PORT/version exposed it as a dev artifact instantly. Our best hypothesis for the slowdown: with a bogus version stamp, version-gated integrations (FlashInfer wheel matching, autotune caches, compat shims) can’t resolve, and the build silently takes fallback kernel paths — the uniform ~2× hit across both prefill and decode fits systemic fallback, not one bad kernel. We didn’t profile further; a mis-stamped build is disqualified either way.
  • Pin nightly-<gitsha> or a digest, never a moving tag. We froze our production image as a locally-tagged derivative so upstream tag drift can’t change what’s running.
  • Smoke-test the exact image before deploying: boot the model on a scratch port, hit the raw /v1/completions endpoint (bypasses chat template and reasoning parser), and watch for both the loud failure (no kernel image is available, cutlass FP4 gemm … Error Internal) and the quiet one (silent garbage tokens).
  • A reasoning model returning empty output is usually not broken. If finish_reason: length and both content and reasoning_content are empty, the model hit max_tokens inside an unclosed <think> block. Raise the budget or pass chat_template_kwargs: {"enable_thinking": false} before you blame the kernels.

The MTP saga: speculative decoding that survives real traffic

Qwen3.6 ships in-checkpoint MTP heads, and single-stream decode on a bandwidth-bound box is exactly where speculative decoding pays. Getting it to stay up taught us the most expensive lesson in this post:

Batch-1 validation lies. Our first MTP deployment (W4A16 checkpoint → Marlin experts, drafter forced to a Triton MoE backend) validated clean at batch-1 with 86.8% acceptance — then died in production with CUDA error: an illegal memory access only under concurrent load, taking both the coordinator tool-calling endpoint and the memory-extraction alias down with it. Upstream issues existed (#41190/#42084); an image bump didn’t fix it; we ran months with spec-decode off.

The crash was avoided by configuration, not a patch. Moving to the W4A4 build routed both the experts and the drafter through FLASHINFER_CUTLASS — a different kernel path than the crashing Marlin+Triton combination. The W4A4/CUTLASS configuration eliminated the crash in our concurrency soak, but checkpoint format and drafter backend changed together, so the result cannot be attributed to either change alone. Before re-enabling in production we soaked it at 8-way concurrency: 400/400 requests, zero illegal-memory-access, 73% acceptance under load.

What MTP actually delivers on GB10 (our production numbers, num_speculative_tokens: 3):

  • Mean acceptance length ≈3.1 output tokens per verification step under 8-way burst — note vLLM’s metric includes the guaranteed bonus token, so with n=3 that’s ~2.1 accepted draft tokens per step (per-position draft acceptance 84.2% / 67.8% / 57.5% — the falling marginal value is why you keep n at 1–3, unlike block-verify methods that run 7–15).
  • ~94.8% cumulative acceptance over ~20 h of real coordinator + extraction traffic — far above any synthetic number, because real agent workloads are predictable, structured output. Measure acceptance on your traffic via /metrics | grep -i spec_decode, not on a demo prompt.
  • Thinking-mode decode is faster than answer-mode with MTP on — structured reasoning accepts better than free prose.

Measured: native FP4 + MTP vs the published Marlin numbers

We ran the same single-stream methodology the community playbook published (prefix-cache-defeating nonces, forced full-length generation, server-side token counts, 3-run medians, ~6K-token prompt) against our production 35B — native FLASHINFER_CUTLASS, W4A4, MTP-3, vLLM 0.23.1rc1-era, measured 2026-07-19:

Config (same model, Qwen3.6-35B-A3B NVFP4) TTFT Prefill Decode (answer) Decode (thinking)
Ours: W4A4 → native FP4, MTP-3, live prod box 955 ms 6,307 t/s 82.2 t/s (79–98 across runs) 102.5 t/s
Published: W4A16 → Marlin, MTP-3, idle box (vlaicu.io) ~965 ms ~6,240 t/s 102 t/s 117.4 t/s (6K ctx)
Ours, historical: W4A16 → Marlin, MTP-3 (2026-06-06) ~91 ms @ short prompt — ~102.7 t/s —

The honest read: these measurements show no batch-1 native-FP4 decode advantage — and they don’t isolate the backend, either. Checkpoint format, machine load, measurement date, and tooling all differ across the rows, so this is convergent evidence, not a controlled experiment. It is consistent with batch-1 parity — decode is memory-bandwidth-bound and 4-bit weight storage is identical either way; our own W4A16-era measurement (102.7 t/s) matches the published Marlin number almost exactly, and our W4A4 medians land lower mostly because the box was serving background production traffic during the run (top of spread 97.6; window acceptance 76.4% vs ~80% idle). Calling it backend equivalence would need a same-checkpoint, same-load A/B we haven’t run. Native FP4’s real wins are prefill/concurrency (compute-bound paths), full cudagraphs on the hybrid arch, and — decisively for us — the kernel path on which MTP stopped crashing. The practical takeaway: don’t switch backends expecting a batch-1 decode win, and when evaluating Marlin, look at prefill and concurrent load — that’s where the difference actually lives.

Patterns from this configuration (one model, one serving config — corroborated where noted, not universal laws): prefill held ~6.3K t/s in both modes on this Qwen3.6-35B setup (TTFT ≈ prompt_tokens ÷ 6300; the vlaicu measurement of the same model reports the same ~6.2K, and note our Nemotron prefills at 8.5K — ‘flat across modes’ generalizes, the absolute number doesn’t); thinking decoded ~25% faster than answering under MTP for this workload and acceptance profile (102.5 vs 82.2); per-position draft acceptance decayed 86.6% → 77.2% → 65.5% across positions 0/1/2 in the same window.

And the no-speculator baseline, same script, same day — our Nemotron-3-Nano-30B-A3B-NVFP4 (native FP4, no spec-decode, vLLM 0.22.1rc1-era):

Metric Answer-only Thinking
TTFT (~6K prompt) 706 ms 710 ms
Prefill 8,530 t/s 8,485 t/s
Decode 64.2 t/s 63.8 t/s

Two things worth noticing: thinking ≈ answering exactly as bandwidth-bound theory predicts when there’s no draft head (the Qwen thinking-speedup is entirely MTP), and the Mamba-hybrid prefills ~35% faster than the attention-hybrid Qwen (8.5K vs 6.3K t/s). Also 64.2 t/s is ~10% above our own 2026-06-02 measurement of the same deployment (58.5) — measured with a different tool; methodology moves numbers, which is exactly why every figure here states its method and date.

Two spec-decode paths we evaluated but do not run:

  • n-gram lookup — the best story in this post, because we got it wrong in public first. Our initial A/B showed n-gram losing 14% on our extraction workload despite 62% acceptance, and we wrote it up as “the free flag isn’t free.” A reviewer pushed back: our explanation blended three effects, and the prefill penalty looked configuration-specific. The controlled retest (every arm with an explicit --max-num-batched-tokens, prefix caching pinned off, k swept 1→4, token-equality checks) flipped the verdict entirely: extract-workload decode went 61.5 → 117.3 tok/s (+91%) at k=3, with no prefill penalty in any arm. The original loss was our configuration all along — an unset scheduler budget interacting with speculation settings — exactly what the boot warning had been trying to tell us. The k-sweep shows the real economics: acceptance decays with depth (80.8% at k=1 → 64% at k=4) while throughput saturates at k=3. Two caveats survived the retest: speculative and plain decode are not token-identical at temperature 0 on this build (every spec arm diverged from the control’s greedy output at the same early position — deterministic batch-shape numerics, so validate output quality, don’t assume equivalence), and all of this is single-stream — speculation taxes throughput under concurrent load, so it ships with --speculative-disable-by-batch-size or not at all. The meta-lesson outranks the numbers: a configuration confound can invert your benchmark’s conclusion, and the warning that would have told you is in the boot log.

    And then the story turned again. Before promoting, we ran the production gates — an 8-way soak plus an output-quality gate with a control arm — and the quality gate killed it: the no-spec control produced 20/20 perfect JSON extractions, while ngram produced 0/20, with version strings losing their final component ("0.22" for "0.22.0") and JSON keys mutating mid-document ("sentence" → "sent"). The GPU proposer failed identically, localizing the defect to the shared verification path. The temperature-0 divergence we’d politely called “batch-shape numerics” was token corruption all along. (Two more gate catches on the way: prefix caching × ngram hard-asserts in the Mamba-2 layers on this arch, and the burst-protection flag most guides recommend doesn’t exist on this build in either spelling.) Final ledger for n-gram on this box: +91% throughput, mechanically real, generating subtly wrong tokens — worthless. Three verdicts in three days, each overturned by a better-controlled experiment than the last. If there’s one sentence to take from this post, it’s the last gate’s: speculative-decoding throughput without output validation is not a result.

  • DFlash — block-diffusion drafters, the biggest published single-stream wins (2.2–2.7× on GB10, per AEON-7’s measurements). The caveat that keeps it off our fleet: on quantized (NVFP4/FP8) targets under stock vLLM, acceptance collapses (~4 of 15 tokens → a net slowdown); the working paths need a patched sm121 build or a BF16 target, and DFlash requires BF16 KV — incompatible with our fp8-KV recipes. MTP at ~95% real-traffic acceptance with zero extra memory sets a high bar.

Unified memory: the OOM you get with “free” RAM showing

--gpu-memory-utilization on Spark is a fraction of the whole 128 GB pool — shared with the OS, page cache, and every other container. Two facts make it dangerous:

  1. The reservation is unswappable. On unified memory, vLLM’s KV-pool grab can’t be paged out. Set 0.6 on a box that’s also running a gateway and observability stack and you’ve made the kernel OOM-killer’s decision for it — that exact misconfiguration cost us a 2h40m production outage (instance reserved ~71 GiB, host hit ~2 GiB free with swap exhausted, oom_kill).
  2. Lowering it shrinks KV capacity, not speed. The pool is capacity, not a performance knob. Our 35B runs at 0.3 (≈34 GiB) while keeping the model’s full 262K context — because the math works:
KV cost  ≈ 12.4 KB/token          ← boot log: 6.78 GiB ÷ 584K pool tokens
                                     (Qwen3.6-35B-A3B, KV cache in fp8)
1 × 262K sequence ≈ 3.1 GiB
0.3 util → 9.9 GiB pool → 3.19 concurrent full-context sequences
real coordinator/extraction traffic touches < 2% of that

A clarification that trips people up: two quantizations coexist in this server and they are different knobs. The weights are NVFP4 (that’s the checkpoint format the whole kernel-selection story is about); the KV cache is fp8 because we pass --kv-cache-dtype fp8. The 12.4 KB/token figure is a property of the KV dtype and the model’s attention layout — the weight quantization plays no part in it. (A widely-quoted figure for this same model family is ~16.6 KB/token, measured on a different checkpoint, vLLM build, and config. Hybrid attention/SSM models are exactly where per-token KV accounting varies across versions — another reason the formula beats the folklore: your own boot log prints both numbers; divide them.)

Read Available KV cache memory and GPU KV cache size ... tokens from your own boot log, and size gpu-memory-utilization from max-num-seqs × max-model-len plus slack — not from a recipe’s 0.85 default, which assumes the box serves nothing else. If a load fails with memory apparently free, NVIDIA’s documented debugging workaround for UMA memory-reporting is to flush the page cache (sync; echo 3 > /proc/sys/vm/drop_caches) and then restart the affected application — the cache holds unified memory CUDA can’t reclaim. Treat it as a diagnostic step, not routine ops.

Smaller sharp edges, quickly

  • Tool calling fails closed, loudly, and only at request time. Without --enable-auto-tool-choice --tool-call-parser <family>, the model loads fine and /v1/models is green — then every agent request 400s ("auto" tool choice requires…). The parser tracks the model family, not your client: qwen3_coder/qwen3_xml for Qwen3.6, nemotron_v3/nano parsers for Nemotron. The wrong parser fails open: the call text sits in content, tool_calls stays null, and your agent silently stalls after the think block.
  • Hybrid-Mamba graph capture (nemotron_h): needed --enforce-eager on v0.20.0 (~37% throughput tax) — fixed by v0.22.0; drop the flag and take the free speed.
  • max_tokens starvation on reasoning models breaks tool-call parsing in ways that look like model bugs. Budget ~10K+ output tokens for reasoning models in agent pipelines.
  • ConnectX-7 clustering: the 200 Gb/s per QSFP cable arrives as two ~100 Gb/s interfaces — configure both or you get half. Each port exposes two RoCE “twins” (each behind an x4 PCIe Gen5 link); our single-interface ib_write_bw measured ~111 Gb/s, consistent with one twin, so full NCCL bandwidth needs NCCL_IB_HCA=<twin1>,<twin2>. DGX Spark also doesn’t support GPUDirect RDMA — CUDA-memory traffic takes NVIDIA’s documented host-pinned fallback. And multi-node vLLM needs more than NCCL: a Ray cluster (or multi-node multiprocessing) with matching runtime environments and model paths, plus an explicitly selected distributed executor. TP=2 across two one-GPU Sparks works; PP=2 can reduce cross-node tensor-parallel collectives where the model supports it.
  • Cold start is real: ~2.5 min weight load + ~25 s first-request JIT on our 30B-class models. Keep instances warm (no idle-timeout, no on-demand start in the request path) and fire a max_tokens: 3 warm-up ping at boot.

What we’d tell you to deploy today

For a 30–35B-class MoE NVFP4 reasoning model on one GB10, July 2026:

  1. Checkpoint: a genuine W4A4 build (check config.json for nvfp4-pack-quantized with activation num_bits: 4; confirm lm_head is in the quant ignore-list if you do structured extraction — 4-bit lm_head gave us !!!! garbage on greedy JSON output).
  2. Image: a cu130-line nightly pinned by gitsha (ours: 0.23.1rc1-era), or v0.25.1 stable after an sm_121 smoke test.
  3. Flags: --moe-backend auto (verify the boot log says FLASHINFER_CUTLASS), --kv-cache-dtype fp8, MTP at num_speculative_tokens: 3 after an 8-way soak, tool-call + reasoning parsers matched to the model family, and --gpu-memory-utilization computed from the KV math above — not defaulted.
  4. Process: pin digests, smoke-test raw completions on every image change, watch /metrics spec-decode acceptance and KV utilization, and never validate spec-decode at batch-1.

Still open: what we’re waiting on, and why it matters

A snapshot of the upstream gaps that shape our config today. Each of these is a reason some recommendation above reads “pinned nightly” or “re-test later” instead of “just do X.”

  • vLLM #41190 — illegal memory access in MTP speculative decoding (open). The crash that took our 35B down for a month. Failure signature: the drafter path poisons the accepted-token buffer (all -1) and the engine dies with CUDA error: an illegal memory access — only under concurrent load, which is why batch-1 validation sails past it. Why it’s pertinent: we didn’t fix it, we routed around it — the W4A4/CUTLASS configuration avoids the crashing kernel combination. Anyone running a W4A16 NVFP4 checkpoint (the Marlin path) with MTP under real concurrency is still exposed. Until it closes, our rule stands: no spec-decode config goes to production without an 8-way soak.
  • vLLM #42084 — out-of-bounds gather with prefix caching + spec decode on sm_121a (open). Constrains which feature combinations are safe on this architecture. Pertinent because prefix caching and speculation are both individually attractive for agent workloads — this is the reason we validate them together, not separately, before enabling both.
  • The b12x backend is deliberately excluded from auto selection pending an upstream CUTLASS sm_121 MMA-guard fix (the exclusion is a documented choice in vLLM’s NVFP4 oracle). b12x (#40082) is the purpose-built sm_121 path and plausibly the fastest one — meaning some performance is intentionally left on the table today. When that guard fix lands, auto may silently start selecting b12x, which is both good news and a behavior change to catch in the boot log. We re-bench on every image bump partly for this.
  • No aarch64 stable wheel has shipped sm_121 cubins in three consecutive releases (v0.22.0 → v0.25.1, verified by us on the day each mattered), and the first CUDA-13 aarch64 “stable” tag turned out to be a mis-stamped dev artifact. This is the single reason our production images are digest-pinned nightlies — not a preference for living dangerously, but the only track whose wheels currently carry the kernels. The day a properly version-stamped cu13 aarch64 stable appears (check /version first), the calculus flips.
  • NGC container lag (vllm #31424). NVIDIA’s official vLLM containers trail upstream by months — the newest NGC tag ships 0.22.1 while upstream stable is 0.25.1. Pertinent because “use the vendor container” is the default advice in most enterprise settings, and on this platform it currently can’t serve the newest model architectures or fixes.
  • Speculative decoding’s scheduler coupling — on this build, CPU n-gram speculation disables async scheduling, and the per-step token budget derives from your batch/speculation settings (vLLM says both in boot warnings). Mostly mitigable from your side: our controlled retest showed that simply setting --max-num-batched-tokens explicitly eliminated the penalty that sank our first experiment — and ngram_gpu (present and working on our 0.22-era build) didn’t disable async scheduling at all. What remains open upstream — now upgraded from observation to confirmed defect: n-gram speculation on this build corrupts structured output (control 20/20 clean vs ngram 0/20; dropped token components inside version strings, mutated JSON keys; identical signature from CPU and GPU proposers → shared verify-path bug), and prefix caching × ngram hard-asserts in mamba_mixer2 on the hybrid arch. Before filing, we duplicate-checked the tracker — and both bugs were already known, which is its own lesson: the assert is precisely open PR vllm#46424 (our data extends it from MTP to ngram), and the corruption belongs to the vllm#39273 hybrid-corruption cluster. We also tested vllm#40875’s config-only workaround (prompt_lookup_min=8): it did not mitigate on this Mamba2 hybrid — corruption changed shape but persisted, 0/20. Our contribution upstream is corroborating data on existing issues, not new tickets — and until that cluster resolves, no n-gram config passes our quality gate regardless of its throughput.
  • One we owe upstream: in our n-gram A/B, temperature-0 output lengths diverged run-to-run with speculation on (502–1116 tokens) while the control was deterministic (529 every run). Speculative decoding is supposed to preserve the target distribution; numerics on this path apparently aren’t bit-stable on this build. We haven’t filed a reproducer yet — until then treat it as our observation, not a confirmed upstream bug.

Appendix: every knob this post mentions

Flag What it actually controls If you get it wrong
--gpu-memory-utilization Fraction of the (unified!) pool pre-allocated for weights + KV Too high on a shared box = unswappable reservation = OOM-killer (our 2h40m outage)
--max-model-len Max prompt+output tokens per request; sizes one sequence’s KV claim Model-card maxima can reserve absurd KV on 128 GB — compute from the KV math
--max-num-seqs Concurrent sequences scheduled High values tax bandwidth-bound decode; also interacts with cudagraph capture sizes
--quantization Overrides quant-format detection (modelopt for nvidia ModelOpt checkpoints) Compressed-tensors checkpoints auto-detect — passing the flag wrongly breaks loading
--moe-backend / --linear-backend Kernel backend for MoE / dense layers (auto = let the oracle pick) Forcing the wrong one: crashes or silent garbage; auto on a cubin-less wheel: no kernel image
--kv-cache-dtype fp8 Halves KV cost vs fp16 Model-specific; some models loop or degrade — validate per checkpoint
--speculative-config Enables spec decoding (method, drafter, k) Unsoaked configs crash under load (#41190); n-gram silently changes scheduler behavior
--enable-auto-tool-choice + --tool-call-parser Lets agent clients send tool_choice: auto; parser must match the model family Missing: every agent request 400s. Wrong parser: calls stall silently in content
--reasoning-parser Splits <think> into reasoning_content Missing: chain-of-thought leaks into the visible reply
--enforce-eager Disables CUDA graphs Was a needed workaround (hybrid-Mamba, ≤v0.20); now just a ~37% decode tax — drop it
--enable-prefix-caching KV reuse for repeated prompt prefixes On by default in V1, but some archs/features disable it — check, don’t assume (see #42084)
--enable-chunked-prefill Interleaves prefill chunks with decode Community-reported ~9× MoE slowdown on some SSM+MoE combos — A/B per model
--attention-backend Pins the attention kernel Forcing one crashed hybrid models on some builds; current nightlies handle flashinfer fine — test per image
--mamba-ssm-cache-dtype SSM state cache precision on hybrid-Mamba models float32 is the validated setting in our Nemotron recipe

Sources & receipts