The GPU isn't the bottleneck: why LLM serving is a memory problem

There’s a stubborn intuition that serving a large language model is about raw compute — buy a bigger GPU, get more tokens per second. For the generation phase, that intuition is mostly wrong. Each new token does a modest amount of matrix math but has to read the entire KV cache — the stored keys and values of every previous token — out of GPU memory. The bottleneck is memory bandwidth and memory capacity, not FLOPs. Once you internalize that, the entire economics of LLM serving rearranges itself around one question: how many requests can you pack into the GPU’s memory at once?

The KV cache is the whole story

When an LLM generates text, it does so one token at a time, and naive attention would recompute the keys and values for every earlier token at every step. The fix — covered in the KV cache lesson — is to cache those K and V tensors so each step only computes the new token’s. That turns per-step cost from quadratic to linear, but it moves the pressure somewhere else: the cache now lives in GPU memory for the lifetime of the request, and it’s large. For long contexts and big batches, the KV cache dwarfs the model weights, and that is what caps how many users you can serve.

PagedAttention: stop wasting memory you reserved

The original way to store the cache reserved a single contiguous block sized for the maximum possible sequence length, per request. A request that ends up using 40 tokens still locked memory for thousands. The result was brutal: traditional KV-cache allocation wasted 60–80% of GPU memory to fragmentation and reserved-but-unused slots.

PagedAttention, introduced at SOSP 2023 and inspired by operating-system virtual memory, splits the cache into small fixed-size blocks handed out on demand — exactly like paging. The effect is dramatic: memory waste drops to under 4%, which means far more sequences fit on the same card. In the original benchmarks, vLLM improved throughput by 2–4× over the previous state of the art at the same latency — purely by not wasting memory.

Continuous batching: never wait for the slowest request

The second half of the win is scheduling. Static batching groups requests, runs them together, and can’t admit a new one until the whole batch finishes — so short requests sit idle behind the longest one. Continuous batching (iteration-level scheduling, first introduced by Orca) swaps requests in and out at the token level, so the moment a sequence finishes its slot is reused. The result is counter- intuitive: continuous batching delivered up to 23× the throughput of naive serving while reducing median latency.

Prefix reuse: the next lever

Once the cache is paged, you can share it. SGLang’s RadixAttention stores the KV cache in a radix tree so common prompt prefixes are reused automatically, with no manual configuration — a big win for chat and agent workloads where every request shares a long system prompt. On prefix-heavy traffic that prefix reuse has been measured at around 6.4× higher throughput, and in head-to-head H100 benchmarks SGLang reached ~16,200 tokens/sec versus vLLM’s ~12,500 — about 29% faster with lower time-to-first-token.

The headline is worth repeating: LLM generation is memory-bound. The teams that serve cheaply aren’t the ones with the most FLOPs — they’re the ones who waste the least KV-cache memory. If you want the mechanism in depth, the KV cache & continuous batching lesson builds it up from the attention math that makes the cache necessary in the first place.