KV cache & continuous batching

Your serving lessons — self-hosting, load balancing, caching, speculative decoding — all quietly rest on one mechanism. It’s the reason LLM inference is bottlenecked by memory, not raw compute, and the reason vLLM exists. Meet the KV cache.

Why the cache exists

Generation is autoregressive: the model produces one token, appends it, and runs again. Naively, every new token would re-run attention over the entire sequence from scratch — recomputing the keys and values for every previous token, every step. That’s quadratic and wasteful, because those keys and values don’t change.

So we cache them. After processing a token, store its attention keys and values; on the next step, only the new token is processed and it attends to the cached K/V of everything before it. That’s the KV cache, and it turns each generation step from “reprocess everything” into “process one token.” The catch: that cache has to live in GPU memory for the whole request, and it’s big.

The KV cache dominates memory

For a real model, the KV cache grows with batch size × sequence length × layers × heads — and it quickly dwarfs the model weights for long contexts. GPU memory, not FLOPs, is what limits how many requests you can serve at once. Which makes how you manage that memory the whole ballgame.

PagedAttention: stop reserving memory you won’t use

The naive approach reserves the maximum possible sequence length for every request up front, as one contiguous block. A request that ends up using 40 tokens still locks memory for 2,000 — so the pool fills with empty reservations and only a handful of requests fit. PagedAttention (the idea behind vLLM) borrows from operating-system virtual memory: split the cache into small fixed blocks and hand them out on demand as each sequence grows. No giant up-front reservation, almost no waste. See the difference on one 48-block pool:

PagedAttention cut KV-cache waste so dramatically that achievable batch sizes — and therefore throughput — jumped severalfold, with near-zero fragmentation.

Continuous batching: never wait for the slowest request

The second half of the win is scheduling. Static batching groups requests, runs them together, and can’t admit a new one until the whole batch finishes — so short requests sit idle waiting for the longest one, wasting the GPU. Continuous batching (a.k.a. iteration-level scheduling) works at the granularity of a single token step: the moment any sequence finishes, it frees its blocks and a waiting request takes its place — mid-flight. The GPU never idles waiting for stragglers.

POOL = 48           # total KV blocks on the GPU
MAXLEN = 12         # max sequence reserved per request (contiguous mode)
req_lens = [4, 3, 6, 2, 5, 3, 7, 2, 4, 3]   # blocks each request actually uses

# Contiguous: reserve MAXLEN per request -> few fit, lots wasted
slots = POOL // MAXLEN
contig_fit = min(len(req_lens), slots)
contig_used = sum(req_lens[:contig_fit])
contig_reserved = contig_fit * MAXLEN

# Paged: allocate only what's used -> pack many more in
paged_fit, ptr = 0, 0
for L in req_lens:
    if ptr + L > POOL: break
    ptr += L; paged_fit += 1

print(f"contiguous: {contig_fit} requests fit, "
      f"{100*(contig_reserved-contig_used)//POOL}% of pool wasted")
print(f"paged:      {paged_fit} requests fit, ~0% wasted")
print(f"\nsame GPU, {paged_fit/contig_fit:.1f}x more concurrent requests with paging")

contiguous: 4 requests fit, 68% of pool wasted
paged:      10 requests fit, ~0% wasted

same GPU, 2.5x more concurrent requests with paging

The whole 48-block pool fills with just 4 contiguous reservations — each locks 12 blocks but uses only a handful, so 68% sits empty-but-reserved. Paging hands out blocks as each sequence actually grows, packing all 10 requests into the same pool. Same GPU, 2.5× the concurrency — and that multiplier is exactly the throughput vLLM buys you.

In one breath

• Generation is autoregressive, so the KV cache stores past tokens’ attention keys/values — each step processes only the new token instead of recomputing everything.
• That cache lives in GPU memory for the whole request and grows with batch × length × layers × heads, so serving is memory-bound, not compute-bound.
• PagedAttention ditches max-length reservation for small on-demand blocks (OS-style paging), nearly eliminating waste and multiplying batch size.
• Continuous batching schedules per token-step: the instant a sequence finishes it frees its blocks and a waiting request slots in, so the GPU never idles on the slowest request.
• Together they are what “use vLLM/SGLang” buys — roughly 10–24× throughput over naive batching at the same latency.

Quick check

Next

The KV cache is the foundation under the cost and throughput numbers in cost & latency engineering, and the reason model routing and caching pay off.