How vLLM actually serves a 7B model

You send a prompt to an LLM endpoint and a few hundred milliseconds later tokens stream back. Between those two moments sits one of the most carefully engineered pieces of systems software in modern computing. Let’s open it up and follow a single request through vLLM — the engine behind a large share of production LLM traffic — from the moment it arrives to the last token it emits.

Two parts, working in lockstep

Strip an inference engine to its core and you find two responsibilities:

• The scheduler manages incoming requests and decides, every single iteration, which ones get to run.
• The KV-cache manager allocates and tracks the GPU memory blocks that hold each request’s attention keys and values.

They’re joined at the hip. When your request arrives, the scheduler asks the KV-cache manager a blunt question: do we have room? If enough free cache blocks exist, the request is admitted and given the blocks it needs. If not, it waits. Nothing about serving makes sense until you understand that the scarce resource being rationed is KV-cache memory, not compute.

The block budget

Here’s the shape of the constraint. Suppose a modern 7B model on an L4-class GPU is configured with around 5,625 cache blocks, each holding 16 tokens. That’s roughly 90,000 tokens of KV cache to share across everyone connected right now. (The figure assumes a grouped-query-attention model like Qwen- or Llama-3-class 7B, which shrinks KV per token several-fold; an older full-multi-head 7B at fp16 would fit far fewer.) For a single user it’s luxurious. For a hundred concurrent sessions — or one session with a very long document — it evaporates.

Why blocks at all? Because the naive approach — give each request one contiguous slab sized to its maximum possible length — wastes enormous amounts of memory to internal fragmentation. PagedAttention borrows the operating system’s oldest trick: virtual memory. Cache is carved into fixed-size blocks that need not be contiguous; a request grabs blocks as it grows and returns them when it finishes. Utilisation climbs from roughly half to nearly full, which in practice means several times more concurrent users on the same card. (We pull this apart in KV cache & continuous batching.)

Prefill, then decode — two different machines

Once admitted, your request lives two very different lives.

Prefill processes the whole prompt at once. Every token attends to every other; the model runs big, dense matrix multiplications across all transformer layers to build the initial KV cache. This phase is compute-bound — it’s doing a lot of arithmetic, and it’s why time-to-first-token grows with prompt length.

Decode then generates one token at a time. Each new token does a tiny amount of compute but must re-read the entire KV cache to do it. This phase is memory-bound — bottlenecked by how fast the GPU can stream the cache and weights out of memory, not by FLOPs. The asymmetry matters: the two phases stress completely different parts of the hardware, which is why modern stacks increasingly schedule them separately.

Keeping the GPU full: continuous batching

A GPU running one request at a time is a Ferrari in a traffic jam. The job is to batch — but static batching (wait, fill a batch, run it to completion, repeat) stalls constantly: short requests finish early and idle while a straggler runs on, and you can’t admit new work until the whole batch is done.

Continuous batching reschedules at every forward pass. The instant a request emits its end token, it’s evicted and its blocks are freed; a waiting request is admitted in the same step. The batch is a living thing, recomposed iteration by iteration, and the GPU rarely idles. This single change bought serving an order-of-magnitude throughput improvement — the subject of its own essay.

When 90,000 tokens isn’t enough

Eventually demand exceeds the block budget. You do not have to drop sessions. The KV cache is working memory — re-read for every token — but not every session is active every moment. So engines offload the least-active cache down a hierarchy: GPU VRAM (fast, tiny) → CPU RAM over PCIe (bigger, roughly an order of magnitude slower on our L4 — ~300 GB/s HBM vs ~32 GB/s PCIe 4.0, and the gap stretches past 60× on HBM3 datacenter cards) → local SSD (huge, slower still). Networked storage is reserved for cold durability, never live decode. An idle session’s cache is parked lower and promoted back to VRAM the instant the user returns — almost always cheaper than evicting it and re-running prefill over the whole prompt. (Full treatment in KV cache offloading.)

Squeezing more from every pass

The last lever is the decode bottleneck itself. Because decode is memory-bound, the GPU is often waiting — which means there’s spare compute to gamble with. Speculative decoding uses it: a small draft model proposes several tokens, the big model verifies them all in one parallel pass, and a rejection-sampling rule guarantees the output distribution is identical to running the big model alone (with greedy decoding that means token-for-token identical output). Accept three or four of four proposals and you’ve roughly tripled tokens-per-pass for free. Its newer cousins — Medusa, EAGLE, n-gram drafting — chase the same prize without a separate model. (See Speculative Decoding.)

The whole machine

Put it together and a request’s journey reads like this:

1. Arrive → the scheduler asks the KV manager for blocks; admit or queue.
2. Prefill → build the KV cache in paged blocks (compute-bound).
3. Decode → generate tokens, re-reading the cache each step (memory-bound), while continuous batching keeps the GPU saturated with other requests.
4. Pressure → offload idle caches down the memory hierarchy; promote on return.
5. Accelerate → speculative decoding turns spare compute into extra tokens.
6. Finish → emit the end token, free every block instantly for the next request.

None of this is visible from the outside. You see an API that returns text. But the reason that API is fast and affordable enough to exist at all is this quiet, relentless choreography of blocks and batches — an operating system for attention, scheduling a scarce resource across everyone at once. If you’re going to run models in production, this is the machine you’re really operating.