Disaggregated serving (prefill/decode split)

Inference metrics left us with an awkward fact: the two phases of inference want opposite things from the hardware. Prefill is compute-bound — it wants raw math throughput. Decode is memory-bandwidth-bound — it wants fast memory and lots of it. Yet the default server runs both on the same GPU, time-sharing it. That co-location causes two problems, and disaggregated serving is the fix: run prefill and decode on separate pools of GPUs.

The interference problem

When prefill and decode share a GPU, a freshly-arrived request’s prefill — a big, compute-heavy burst — has to be squeezed in between the decode steps of every request already streaming. That burst stalls those decodes: their inter-token latency spikes for one step, and the user sees a visible hitch mid-answer. The more (or longer) the prompts arriving, the more the ongoing answers stutter.

base_itl, prefill_ms = 15, 80
# Co-located: a new request's prefill is injected at step 5, stalling that decode step
colocated = [base_itl] * 10
colocated[5] += prefill_ms
# Disaggregated: prefill runs on a separate pool; the decode stream is untouched
disagg = [base_itl] * 10

print("co-located ITL (ms):", colocated, "-> worst", max(colocated))
print("disaggregated  (ms):", disagg, "-> worst", max(disagg))

co-located ITL (ms): [15, 15, 15, 15, 15, 95, 15, 15, 15, 15] -> worst 95
disaggregated  (ms): [15, 15, 15, 15, 15, 15, 15, 15, 15, 15] -> worst 15

Co-located, the steady 15 ms stream jumps to 95 ms the moment a prefill lands — a p99 ITL six times the median, all from interference. Disaggregated, decode just keeps decoding.

The split

Disaggregation puts prefill and decode in two pools:

1. A request hits a prefill worker, which processes the prompt and builds its KV cache.
2. That KV cache is transferred to a decode worker.
3. The decode worker streams the output tokens, attending to the transferred cache — uninterrupted, because prefills never run here.

What disaggregation buys

• No interference → smooth ITL. Decode workers only ever decode, so the per-token latency stays flat and the p99 tail shrinks. That is the single biggest user-visible win.
• Independent scaling. Prefill load (prompt tokens/sec) and decode load (active streams) rarely move together. Disaggregated, you scale each pool to its own demand — add prefill workers when prompts are long, decode workers when concurrency is high — instead of over-provisioning one GPU type for both.
• Hardware specialization. You can even put the pools on different GPUs: compute-dense parts for prefill, memory-bandwidth-rich parts for decode — matching silicon to bottleneck.
• Better goodput at SLO. Because TTFT (prefill) and ITL (decode) no longer contend, you can hit a tight TTFT and a tight ITL target at once — which a co-located server struggles to do.

The cost: moving the KV cache

Disaggregation is not free — between step 1 and step 3 you must ship the KV cache from the prefill worker to the decode worker, and that cache can be large. So the technique lives and dies on KV transfer cost:

• Fast interconnect (NVLink, InfiniBand, RDMA) makes the transfer cheap enough to be worth it; over slow networking it can erase the gains.
• KV-locality routing. Route a request to a decode worker that already holds a relevant prefix’s KV (or keep related requests on the same worker), so you transfer — or recompute — less. The same idea scales up to multi-region serving: keep a user’s KV near where they are.
• KV offloading. Systems like LMCache tier the KV cache across GPU, CPU, and disk so a large prompt’s cache can be stored and re-served cheaply — pairing naturally with KV-cache offloading.

In one breath

• Prefill (compute-bound) and decode (memory-bound) want opposite hardware, but a co-located server runs both on one GPU, where a new request’s prefill stalls ongoing decodes — an ITL spike (95 ms vs a 15 ms median in the demo).
• Disaggregated serving splits them into a prefill pool (builds the KV cache) and a decode pool (streams tokens), with the KV cache transferred between.
• Wins: smooth ITL (no interference, smaller p99), independent scaling of each pool, hardware specialization, and better goodput at a tight SLO.
• The cost is moving the KV cache, so it depends on fast interconnect and KV-locality routing (and KV offloading/tiering) to keep transfer cheap.
• It composes with PagedAttention + continuous batching and is shipped by vLLM, NVIDIA Dynamo, DistServe/Splitwise, and the big providers.

Quick check

Next

Disaggregation rests on the prefill/decode metrics and on cheap KV movement — see KV-cache offloading for the tiering that makes large caches affordable, and self-hosting for the engines (vLLM, Dynamo) that implement these modes.