For a workload with very long prompts and long streamed responses, when would you split prefill and decode onto different worker pools? How would KV-cache transfer, network bandwidth, scheduling, and KV offloading affect whether disaggregated serving actually improves goodput?
Split prefill and decode when their different resource demands cause harmful interference and the network can move KV caches quickly enough. The decision depends on end-to-end goodput under latency SLOs, not on GPU utilization or raw token throughput alone.
How to think about it
The answer
I would split prefill and decode when long prompts make the initial forward pass bursty and long generations keep GPUs occupied for minutes, so shared workers create head-of-line blocking and violate time-to-first-token or streaming-latency SLOs. I would do it only when that isolation benefit exceeds the cost of moving the prompt’s KV cache across the network and managing two coupled queues; otherwise, a well-scheduled unified pool usually wins.
Why the phases behave differently
Prefill is the first pass over the user’s prompt. The model processes all input tokens and builds a KV cache, the stored keys and values that later attention layers reuse. Decode generates the response one token at a time, consulting that cache instead of recomputing the entire prompt.
Those phases stress hardware differently. Prefill has lots of parallel work across the prompt and is often compute-heavy. Decode has little new computation per sequence but repeatedly reads model weights and a growing KV cache, so it is often constrained by memory bandwidth and per-step latency. Those are common patterns, not laws: very large decode batches can become compute-bound, and hardware changes the balance.
On one shared worker, a 32,000-token prefill can occupy a batch while a streaming response is waiting for its next token. If the scheduler protects the decode batch, the new prompt waits. If it admits the prompt immediately, existing users see a pause between streamed tokens. The conflict is not merely cosmetic. A user may tolerate a two-second first-token delay but notice a 500-millisecond gap in the middle of a sentence.
Separate pools let each phase use its own batching policy. Prefill workers can batch or chunk long prompts. Decode workers can use continuous batching, adding and removing sequences as requests start and finish. The scheduler can also scale the pools independently.
The relevant target is goodput: useful requests completed per unit time while meeting the service-level objective, or SLO. A system that produces more raw tokens per second but causes half its requests to miss their latency targets has improved throughput, not goodput.
The cache-transfer bill
Disaggregation does not send a small handoff message. For a normal decoder-only model, the decode worker needs the KV entries for the prompt at every transformer layer. Sending only the final hidden state or the sampled token is not enough; the destination would have to recompute the prompt.
Consider an illustrative model with:
- 80 transformer layers
- 8 KV heads
- 128 values per head
- BF16 cache entries, at 2 bytes each
- separate K and V values
The cache size per token is:
80 × 8 × 128 × 2 × 2 bytes = 327,680 bytes
That is 320 KiB per token. A 32,768-token prompt therefore produces 10 GiB of KV cache. An 8,192-token response adds another 2.5 GiB, although that response cache is normally created on the decode worker rather than transferred at the initial handoff.
On an ideal 100-gigabit-per-second link, 10 GiB takes roughly 0.86 seconds to transfer. That is the optimistic single-request number. Ten simultaneous handoffs represent about 100 GiB and take roughly 8.6 seconds if they contend for the same aggregate link. Real effective bandwidth is lower because of protocol overhead, concurrent traffic, page management, and possible cache reformatting.
The cache is often paged and sharded across tensor-parallel GPUs. A transfer may therefore involve more than a device-to-device copy: the source and destination must agree on page layout, ownership, and parallelism. Moving from an eight-way tensor-parallel prefill group to a four-way decode group may require repartitioning the cache.
This creates the central trade-off:
| Condition | Why splitting helps | Why splitting hurts |
|---|---|---|
| Long prompts | Removes prefill interference from active streams | Produces a large KV handoff |
| Long responses | Frees prefill workers quickly while decode continues elsewhere | Keeps decode workers occupied for a long time |
| Fast private fabric | Makes handoff latency and contention manageable | An oversubscribed network becomes the new queue |
| High concurrency | Justifies separate pools and independent scaling | Too many simultaneous handoffs can saturate the fabric |
For a response lasting 8,192 tokens, a hypothetical decode rate of 50 tokens per second means the worker is occupied for about 164 seconds. In that case, a one-time 0.86-second transfer may be small compared with total service time, but it can still be unacceptable if the first-token SLO is 300 milliseconds. Whether the transfer is “cheap” depends on the SLO, not just on the fraction of total generation time.
Scheduling determines whether the architecture works
A production scheduler should reserve decode capacity before launching expensive prefill work. Otherwise, it can build a 10-GiB cache and then leave it waiting in a transfer queue because every decode worker is full. The result is wasted GPU work plus terrible time to first token.
Scheduling by request count is also misleading. One 1,000-token prompt and one 32,000-token prompt are not equivalent prefill jobs. One 200-token answer and one 8,192-token stream are not equivalent decode jobs. Admission and routing should account for estimated prompt tokens, expected output tokens, KV bytes, available HBM, and current network load.
A useful production pattern is:
- Admit the request only when a decode worker has cache capacity and a plausible start time.
- Run prefill with chunking or batching so one huge prompt does not block every other prompt.
- Transfer KV pages over a bandwidth-aware path.
- Place the request into continuous decode batching.
- Apply backpressure when the decode pool or transfer fabric is full.
This also means the two pools cannot be autoscaled independently in a simplistic way. If prefill capacity doubles while decode capacity does not, the system merely manufactures a larger handoff queue. The right pool ratio comes from traffic histograms and measured queueing, not from an attractive one-to-one GPU ratio.
KV offloading can change the answer
KV offloading means moving some cache from GPU HBM, the GPU’s high-bandwidth memory, to another tier such as host DRAM or NVMe. It increases the number of sessions the decode pool can hold, which is valuable when a 32,000-token prompt consumes 10 GiB before the answer has even grown.
But offloading does not make the bytes disappear. It exchanges HBM pressure for PCIe, NVLink, or storage traffic. If hot KV pages are repeatedly pulled from host memory during decode, the token loop can stall. The symptom is usually a sudden rise in inter-token latency, falling decode-GPU utilization, and a saturated host-device link. NVMe is generally suitable for cold or parked sessions, not for cache pages needed on every active token step.
Offloading can also make disaggregation worse. A prefill worker may create KV in GPU memory, spill it to host memory, and then send it over the network to the decode worker. That adds another copy and another bandwidth bottleneck. Keeping active pages in HBM, offloading inactive sessions, and transferring only once is a much healthier design.
KV compression or lower-precision cache formats can reduce the handoff size, but they add kernel and quality considerations. They need end-to-end measurement; a smaller cache is not automatically a faster cache.
The failure mode to watch for
The classic failure looks good in averages and awful at the edge: median throughput rises, but p99 time to first token and inter-token latency spike. Network interfaces sit near line rate, decode GPUs show idle gaps while waiting for cache pages, and the prefill queue keeps growing.
That usually means the system optimized the compute pools but forgot the transfer queue. Fix it with transfer-aware admission, decode-capacity reservation, token- and byte-based load balancing, and explicit backpressure. If the cache transfer is slower than simply running prefill again on the destination, do not transfer it. Recompute can be the less absurd choice.
What they’ll ask next
Why not keep one pool and use continuous batching?
Often, that is the right answer. A unified pool avoids KV transfer and duplicated model weights. Split only when phase interference is measurably harming SLO-weighted goodput, even after chunked prefill and a competent continuous-batching scheduler.
Can the prefill worker send only the final hidden state?
Not for ordinary decoder-only generation. Each new token attends to the prior tokens’ keys and values at every layer, so the destination needs the KV cache or must rerun the prompt.
What would you measure in an experiment?
Compare unified and disaggregated serving at the same model, hardware budget, traffic mix, and SLOs. Measure goodput, p50 and p99 time to first token, inter-token latency, completion time, KV bytes transferred, network utilization, HBM occupancy, offload traffic, and queue wait. Raw tokens per second is not enough.
One line to say in the room
“I split prefill and decode only when their interference hurts SLO-weighted goodput, and I prove the network, cache capacity, and scheduler can move the KV handoff without simply replacing GPU queueing with network queueing.”