How vLLM actually serves a 7B model

A request can be perfectly valid, the model can be loaded, and the GPU can still spend its evening making users wait.

The usual culprit is not the neural network in isolation. It is the machinery around it: deciding which requests run now, finding memory for their growing attention state, and preventing one enormous prompt from holding every other request hostage.

That is what vLLM is really doing. The HTTP endpoint is the easy part.

Follow one request through the engine and a useful rule emerges: online LLM serving is primarily a problem of scheduling scarce KV-cache memory. The matrix multiplications matter, of course.

But if you misunderstand the cache and the scheduler, you will tune the wrong knobs, misread GPU utilisation, and eventually discover that “90,000 tokens of capacity” does not mean 90,000 useful tokens for every workload.

Start with one request

Imagine Northstar, an internal support assistant running a 7B model on one NVIDIA L4 with 24 GB of memory.

A typical request contains a long system prompt, retrieved policy documents, and the conversation so far: 12,000 input tokens. The product allows up to 200 generated tokens.

Thirty employees may submit requests at roughly the same time. That sounds like a language problem. To the serving engine, it is a memory problem.

The model weights occupy memory before the first request arrives. The request then needs more memory for the KV cache, the saved key and value vectors that let future tokens attend to earlier tokens without recomputing them.

Every token in every live sequence adds to that cache. Thirty long requests can exhaust memory even though the model weights never change.

The useful mental model has two cooperating responsibilities:

• The scheduler chooses which sequences run during each model iteration and how many prompt tokens each one may process.
• The KV-cache manager assigns physical memory blocks to those sequences, tracks which blocks are free, and dereferences them when a sequence finishes.

These are not necessarily two separate processes. They are two jobs inside a larger runtime that also includes tokenisation, model workers, CUDA kernels, communication, and request handling.

But this split explains most of the behaviour you see from the outside.

A 7B label tells you almost nothing about cache capacity

People often estimate memory from parameter count alone. A 7B model in fp16 needs roughly 14 GB for its weights because each parameter takes about 2 bytes.

On a 24 GB L4, that leaves only a limited remainder for CUDA workspaces, activations, runtime state, and KV cache.

The cache size depends on the model’s architecture, not just its parameter count. For one token, a simplified fp16 calculation is:

bytes_per_token = 2 × layers × KV_heads × head_dim × 2

The first factor of 2 accounts for keys and values. The last factor of 2 is the number of bytes in an fp16 value.

Take a Qwen-like grouped-query attention model with 28 layers, 4 key/value heads, and a head dimension of 128:

bytes_per_token = 2 × 28 × 4 × 128 × 2
                 = 57,344 bytes

That is about 56 KiB of KV cache per token. A 16-token block therefore needs 917,504 bytes, just under 0.9 MiB.

If the runtime has 5,625 such blocks, the arithmetic is:

5,625 × 16 = 90,000 token slots

The cache itself is about 5.16 GB in this example. That is where the familiar “roughly 90,000 tokens on an L4” figure can come from.

It is an illustrative configured budget, not a property of every 7B model. The number of blocks is measured after the runtime reserves memory for weights and other GPU work.

Architecture can move the result dramatically. A Llama 3 8B-style configuration with 32 layers, 8 key/value heads, and the same head dimension uses:

2 × 32 × 8 × 128 × 2 = 131,072 bytes per token

That is more than twice the KV cost per token. Ninety thousand tokens would need about 11.8 GB of cache, before counting roughly 16 GB of fp16 weights for an 8B model.

The same L4 cannot offer the same cache budget in that configuration.

Quantised weights may create more room for KV cache. Quantised KV cache may reduce it further, with its own accuracy and kernel trade-offs.

Longer context is not free. More layers, more KV heads, wider head dimensions, and more live sequences all consume the same scarce resource.

This is why “the model fits” is only the first memory question. The production question is: how many actual live tokens can fit after the model and runtime have taken their share?

Why vLLM uses blocks

A naive allocator might reserve one contiguous cache region for each request. If a request is allowed to reach 32,000 tokens, it could reserve space for all 32,000 even when it currently contains 700.

With many requests of different lengths, large holes appear between allocations. Some free memory exists, but no single contiguous region is large enough for the next request. That is fragmentation.

vLLM’s PagedAttention uses a different arrangement. It divides the KV cache into fixed-size blocks, much like an operating system divides memory into pages.

A sequence receives a logical block for token positions 0 through 15, another for positions 16 through 31, and so on. Those physical blocks can live anywhere in the GPU’s pool.

The attention kernel follows a block table to find the blocks belonging to a sequence. Logical token order remains intact even when physical addresses are scattered.

When a sequence grows, the manager takes another free block. When a sequence ends, its blocks are dereferenced.

Without prefix caching they return to the free pool. With prefix caching, eligible full blocks may remain as evictable cache entries. Blocks referenced by active sequences cannot be evicted.

The benefit is not magic extra VRAM. Paging prevents the allocator from reserving a giant maximum-sized slab and makes free space reusable across requests.

Only the final partially filled block has ordinary tail waste, at most 15 token positions with a 16-token block size.

Block size is a trade-off. Smaller blocks reduce tail waste and make sharing more precise, but require more metadata and may add kernel overhead.

Larger blocks reduce bookkeeping but waste more space at sequence ends and make prefix sharing coarser.

There is another useful optimisation here. With prefix caching enabled, identical leading tokens can share full KV blocks between requests.

If every Northstar request begins with the same 2,000-token policy prompt, those requests may share 125 complete 16-token blocks instead of storing that prefix repeatedly.

The user-specific part still needs its own blocks, and an unreferenced cached block can be evicted when memory pressure rises. Prefix caching helps only when the tokenised prefix is actually identical; “nearly the same prompt” does not count.

The details of KV memory, prefix reuse, and block allocation are worth keeping separate from the model itself. KV cache mechanics covers that layer in more depth.

What happens when Northstar arrives

Northstar’s 12,000-token prompt needs:

12,000 ÷ 16 = 750 blocks

After generating 200 tokens, its sequence contains 12,200 tokens. It needs:

ceil(12,200 ÷ 16) = 763 blocks

The last block contains only 8 tokens.

With 5,625 available blocks, seven such requests need 5,341 blocks. For eight same-length requests with no reusable prefix blocks, eight need 6,104.

So eight identical Northstar requests cannot all keep their complete KV state in this illustrative budget. That is true even though “90,000 tokens” sounds like enough for more than seven requests.

The remaining tokens are not a separate pool. They are the sum of every live sequence’s cache.

That no-sharing assumption matters. If all eight share the stated 2,000-token exact prefix, the calculation changes to 125 shared blocks plus 8 × 638 private blocks for the remaining 10,200 tokens per request, or 5,229 blocks.

That fits the illustrative 5,625-block pool. Prefix caching can change the answer; it does not make every similar prompt shareable.

The scheduler does not simply accept or reject a request once and then disappear. At each scheduling iteration, it looks at running sequences, waiting sequences, available blocks, and a token budget for the next forward pass.

A decoding sequence usually asks for one new token’s worth of cache in an iteration. A new prompt may ask for hundreds or thousands of tokens during prefill.

If the scheduler admits a long prompt, it must reserve enough blocks as that prompt is processed. If there is no safe room, the request waits, or an already-running sequence may be preempted according to the engine’s policy.

Three scheduler and resource limits

There are three scheduler and resource limits to keep distinct:

1. KV-cache blocks limit how many token states can remain resident.
2. The batched-token budget, commonly configured as max_num_batched_tokens, limits how much prompt and decode work the next forward pass may contain.
3. The active-sequence limit, commonly configured as max_num_seqs, limits how many sequences can be active in the batch at once.

A workload can have free KV blocks and room in the batched-token budget but still queue requests when max_num_seqs has been reached.

Conversely, a server can have active-sequence slots available but lack enough blocks for another long request. It may also have enough memory but too many tokens scheduled for one iteration.

There is also a separate per-request limit: the model’s maximum context length and the server’s configured maximum context length, often exposed in vLLM as max_model_len.

Together, they cap the total prompt and generated history that one request may reach. The server setting can impose a lower boundary; it cannot make the model support a longer context than its architecture allows.

Counting requests alone hides all four constraints. “Thirty concurrent requests” is not a useful capacity description unless you also know their prompt and output lengths.

Prefill and decode are different workloads

Once admitted, a request has two distinct phases.

Prefill processes the input prompt and builds KV entries for those tokens. Prompt positions can be processed in parallel, subject to the causal attention pattern, so the GPU can use large matrix operations efficiently.

A 12,000-token prompt creates 12,000 tokens of KV state and usually does substantial arithmetic. Long-prompt prefill is often compute-bound.

The first generated token cannot arrive until enough prefill has completed. Time to first token, usually abbreviated TTFT, therefore includes queue time as well as prompt processing time.

A model can have a fast forward pass and still show awful TTFT because requests are waiting behind other prefills.

Decode generates the response one token at a time. For each new token, the model computes a query and compares it with the keys for all earlier positions, then combines the corresponding values.

The old keys and values are reused; the new token’s entries are appended to the cache.

Decode performs less new arithmetic per sequence than a long prefill, but it repeatedly reads model weights and the growing KV cache from memory. It is therefore often memory-bandwidth-bound.

On the L4, the relevant device memory is GDDR6 with a stated peak bandwidth around 300 GB/s, not HBM. The exact bottleneck changes with batch size, quantisation, context length, and kernel implementation.

A very large decode batch can become compute-limited.

The user-visible metric is different too. Inter-token latency is the time between streamed output tokens.

A long prefill may barely change the average output throughput while making every other user wait for their first token. A decode-heavy workload may have acceptable TTFT but jerky streaming because cache reads or transfers dominate.

That is why one average latency number is nearly useless. Track queue time, TTFT, inter-token latency, prompt tokens, generated tokens, and the 99th percentile, written as p99.

The p99 is the slowest point among roughly the slowest one percent of requests. It is where the 3 a.m. page usually lives. Inference metrics gives these measurements a proper treatment.

Continuous batching keeps the batch alive

Static batching waits for a group of requests, runs them together, and often keeps the batch shape around until the group finishes.

Suppose request A needs 40 output tokens and request B needs 400. Once A finishes, its slot may sit idle while B continues.

A newly arrived request C waits for the whole batch, even though A’s resources are free.

Continuous batching reschedules at every iteration. When A emits its stop token, the scheduler dereferences A’s blocks and can place C into the next iteration.

The active batch is continually rebuilt rather than treated as a fixed group.

This is more than a throughput trick. It changes the unit of scheduling from “one request until completion” to “the next small piece of work.”

That lets short requests finish without waiting behind long ones and lets the GPU combine decode work from many sequences.

There is a catch. A long prefill can still crowd out decode work if the scheduler gives it an enormous token budget.

Chunked prefill splits a long prompt into several pieces so that decode tokens from existing users can be interleaved with those pieces. A 12,000-token prompt might be processed as six 2,000-token chunks rather than one large launch.

Smaller chunks usually protect streaming latency better, but can reduce prompt-processing efficiency through extra scheduling and kernel overhead.

Larger chunks may improve prompt throughput while causing visible pauses for users already receiving tokens. The right value depends on the workload.

“Keep the GPU full” is not the same as “make every request feel fast.”

When the blocks run out

Return to Northstar. If eight long requests arrive together, the illustrative 5,625-block pool cannot hold all 6,104 required blocks.

Something has to give.

The safest answer is often to queue the eighth request. Queuing is boring, explicit, and much better than pretending the GPU has memory it does not have.

An engine may instead preempt a running sequence. One strategy releases its KV blocks and later recomputes the request’s full cached prefix, including any tokens already generated before preemption.

That saves GPU memory but repeats expensive prefill work. Another strategy swaps state to CPU memory when configured and brings it back later.

That avoids recomputation but pays transfer time and consumes host memory.

This is where diagrams showing a smooth VRAM → RAM → SSD ladder become misleading. PagedAttention gives you a block-based GPU allocator.

It does not automatically turn every storage tier into transparent, low-latency cache.

Offload changes the cost

CPU KV offload can be useful for genuinely idle sessions, especially when the alternative is throwing away a large conversation and recomputing it.

But an actively decoding sequence needs its attention state available to the GPU. Promoting a cold cache is a real transfer, not a pointer update.

For Northstar’s 12,000-token Qwen-like prompt, the KV state is about 688 MB. PCIe 4.0 x16 has a theoretical one-way bandwidth around 31.5 GB/s, so moving that state once has a lower-bound transfer time above 20 milliseconds.

That is before software overhead, synchronisation, and contention. Real behaviour is less tidy. Moving or faulting blocks during a streaming response can turn smooth output into bursts.

SSD-backed live KV storage is even less like VRAM. It can be part of a deliberately designed offload system, but it brings storage latency, queueing, wear, and recovery questions.

Network storage is appropriate for durable conversation data, not as though it were a nearby attention cache.

If the product needs cold-session storage, decide whether to restore KV, recompute from the saved transcript, or summarise the conversation. Those are different latency and quality choices. KV-cache offloading examines the trade-off.

Common pressure responses include:

• reducing the maximum context,
• capping generated output,
• using a smaller or quantised model,
• moving to another GPU,
• sharing repeated prefixes, or
• accepting queue time.

None is universally best. The important thing is to choose deliberately instead of allowing an accidental preemption policy to decide for you.

Speculative decoding spends spare compute

Decode is often constrained by memory movement, which can leave some arithmetic capacity unused. Speculative decoding tries to spend that spare compute.

A small draft model proposes several next tokens. The larger target model then verifies those candidates in a single causal pass.

If the candidates are good, the target advances several positions while paying roughly one target-model scheduling round. If a candidate is rejected, the target supplies the correction and later candidates are discarded.

With a correctly implemented acceptance and correction procedure, speculative decoding can preserve the target model’s sampling distribution.

Under greedy decoding, accepted candidates match what the target would have selected, so the final sequence matches target-only greedy generation.

It is not free. The draft model consumes GPU memory and compute. The target still performs work to verify multiple positions.

Acceptance varies with the prompt, model pair, and sampling settings. A draft model that proposes poor continuations adds overhead without advancing many tokens.

Speculative decoding is a lever to measure, not a ceremonial checkbox. The speculative decoding guide explains the acceptance process and its limits.

The failure you will see first

One common incident looks paradoxical: GPU utilisation is high, but streamed responses arrive in bursts and p99 TTFT climbs.

Long prefills are consuming large scheduling slices, so existing decode requests are waiting for their next iteration. Chunking the prefill or adjusting the batch token budget can help, but only after measuring the effect on prompt throughput.

Another failure appears before traffic arrives. The server starts loading the model and runs out of memory at startup.

Identify the failure phase before changing knobs:

• An OOM while loading weights or allocating the KV pool means the weight footprint, runtime reservations, and requested cache budget do not fit. Use a smaller model, lower precision, or a smaller cache budget.
• An OOM during CUDA-graph capture, workspace allocation, or per-batch metadata allocation may instead be fixed by lowering maximum sequences or batched tokens, such as max_num_seqs or max_num_batched_tokens.

Lowering concurrency will not fix a weight-load or KV-pool sizing failure. It may fix these graph, workspace, or metadata allocation failures.

A third symptom is a sudden rise in latency after enabling offload: average throughput looks acceptable, but p99 inter-token latency develops long spikes.

That points to cache promotions, CPU contention, PCIe transfers, or recomputation. More GPU utilisation is not the remedy.

Fewer offloaded live sequences, a smaller context, or a different admission policy may be.

What to do on Monday morning

Measure the workload first

Start with the workload, not a configuration file. Record the distribution of prompt tokens, generated tokens, concurrent live sequences, and repeated prefixes.

Keep the long tail. An average 800-token prompt does not describe a service where one request in twenty contains 12,000 tokens.

Then run three load tests:

• short prompts with short answers,
• long prompts with short answers, and
• a mixed stream that resembles production.

Measure queue time, TTFT, inter-token latency, output tokens per second, free cache blocks, waiting sequences, running sequences, and preemptions.

Test the point where the eighth Northstar-style request arrives, not just the comfortable point where seven fit.

Set deliberate limits

Set product limits from those results. A maximum context length and maximum output length are capacity controls, not merely user-interface settings.

Leave deliberate memory headroom instead of reserving every last block. The exact margin belongs in your load test because workspace requirements and traffic patterns differ.

Protect interactive traffic from long prefills with chunked scheduling. If users care about first-token and streaming latency, do not tune only for aggregate throughput.

Conversely, if the service runs offline document summarisation overnight, larger batches and longer scheduling slices may be the correct choice.

Use caching and offload deliberately

Enable prefix caching when the workload contains stable exact prefixes. Confirm that it reduces prefill work and duplicate blocks under realistic prompts.

Do not expect it to help a chat history that changes on every request.

Treat CPU or SSD offload as a capacity extension with a latency bill. Define which sessions are cold, how they return to GPU memory, and what happens when host memory fills.

If the system cannot answer those questions, it is not an offload strategy yet. It is a future incident.

Measure speculative decoding

Only after this baseline should you test speculative decoding. Compare accepted tokens per verification round, TTFT, inter-token latency, GPU memory use, and total cost.

Disable it if the draft model rarely helps.

When simpler is better

The strongest counterargument is fair: if you run one request at a time, use short fixed prompts, or process a known offline batch, all this machinery can be unnecessary.

A simple model runner may be easier to operate. Paged allocation, continuous scheduling, prefix caching, and speculative decoding earn their complexity when requests overlap, lengths vary, and users expect streaming responses.

But that is not the normal shape of a shared online endpoint. There, the 7B weights are only the resident machinery.

The real workload is a changing population of token histories competing for a finite pool of KV blocks. vLLM’s scheduler and cache manager are the part that turns that competition into a service instead of a queue with a GPU attached.