GPU compute isn't always the bottleneck: why decode can become a memory problem

At 2:17 a.m., your model weights fit comfortably on the GPU. One request is fast. Then traffic arrives: forty chat requests, each with a different history, and the server starts rejecting new work with CUDA out of memory.

Nothing about the model changed. The weights did not suddenly become larger. Each conversation brought its own growing memory object: the KV cache, the stored attention state for every token already processed.

I have made the mistake of treating this as a compute problem. The instinct is natural. Large language models contain billions of parameters, and GPUs are sold by their floating-point operations per second.

So you buy a larger card, or a more expensive one, and expect tokens per second to follow.

That is the wrong first question for interactive generation.

The useful question is: how many active sequences can the GPU hold, and how quickly can it move their cached state through memory?

The GPU still does the work. The bottleneck is often not its arithmetic units. It is the memory system feeding them.

One request contains two very different workloads

Inference, running a trained model to produce an answer, has two phases.

Prefill processes the prompt. If the prompt contains 8,000 tokens, the model can process much of that sequence in parallel. The large matrix multiplications have healthy shapes, so the GPU’s tensor cores often have plenty to do.

Prefill largely determines time to first token, or TTFT: the delay before the first generated token appears.

Decode produces the answer one token at a time. The model generates token 1, then uses token 1 to generate token 2, and so on. Every step has a small new query but must consult the accumulated history.

Decode determines inter-token latency, the delay between visible output tokens, and contributes most of the work for long answers.

That distinction matters because “LLM serving is memory-bound” is mainly a claim about decode.

During prefill, the GPU sees a large batch of token positions and can reuse data efficiently. During decode, it may see one new token per request. The matrices become narrow.

The GPU can have enormous theoretical compute capacity and still spend much of its time waiting for data to arrive from high-bandwidth memory, or HBM.

The same server can therefore be compute-bound for a 20,000-token document upload and memory-bound for a busy chat endpoint.

“The model is compute-bound” is not a property of the model alone. It is a property of the model, phase, batch shape, sequence lengths, and scheduler.

Why the KV cache gets so large

Attention needs three things:

• a query, which asks what the new token should look at
• a key, which describes each earlier token for matching
• a value, which contains the information retrieved after a match

For the current token, the model creates a new query. It compares that query with the keys for all earlier tokens, then uses the resulting weights to combine their values.

The keys and values for old tokens do not change, so the server stores them instead of recomputing them. That stored state is the KV cache.

The cache removes repeated computation, but it must live somewhere. During GPU serving, it usually lives in GPU memory for the lifetime of the request. Every generated token extends it.

For a conventional transformer layer, the raw KV-cache size is approximately:

2 × layers × KV heads × head dimension × sequence length × bytes per value

The first factor of 2 is for keys and values. “KV heads” means the number of key and value heads, which can be smaller than the number of query heads when the model uses grouped-query attention.

“Bytes per value” is typically 2 for FP16 or BF16.

Here is a concrete architecture:

• 80 transformer layers
• 8 KV heads
• head dimension of 128
• BF16 cache, so 2 bytes per value
• 8,192 tokens in the prompt plus generated history

def kv_cache_gib(layers, kv_heads, head_dim, seq_len, bytes_per_value):
    raw_bytes = 2 * layers * kv_heads * head_dim * seq_len * bytes_per_value
    return raw_bytes / (2 ** 30)


per_request = kv_cache_gib(
    layers=80,
    kv_heads=8,
    head_dim=128,
    seq_len=8192,
    bytes_per_value=2,  # BF16 or FP16
)

print(f"{per_request:.2f} GiB")
print(f"{per_request * 32:.2f} GiB")

2.50 GiB
80.00 GiB

That is 2.5 GiB of raw KV cache for one 8,192-token request. Thirty-two such requests need 80 GiB before counting model weights, temporary activations, CUDA workspaces, allocator metadata, or any safety margin.

The example resembles a large grouped-query model, but the formula is architecture-specific. A smaller model with many KV heads can consume more cache per token than a larger model with aggressive grouped-query attention.

The parameter count printed on a model card does not tell you the cache size by itself.

This is why weight quantization often disappoints teams during serving. A 70-billion-parameter model stored at roughly four bits per parameter needs around 35 GB before quantization metadata and runtime overhead.

That can make the weights fit on an 80 GiB card. It does not make a BF16 KV cache four bits wide. If the cache remains BF16, every long conversation still consumes the same cache memory.

Capacity and bandwidth are separate ceilings

The cache creates two separate ceilings:

• Capacity determines how many requests can be resident at once. When there are no free blocks for a new sequence, the scheduler must queue or reject it.
• Bandwidth determines how quickly the resident requests can advance. At each decode step, the attention kernels read the keys and values for the previous tokens. The longer the history and the larger the batch, the more memory traffic they create.

Suppose those sixteen requests from the example all advance by one token. The raw KV traffic associated with that iteration is about 40 GiB.

An H100 SXM advertises 3.35 TB per second of HBM bandwidth, so moving 40 GiB would take roughly 13 milliseconds even in an idealized calculation.

That is not a latency benchmark. It excludes weights, kernels, synchronization, cache effects, and every other operation. It is a lower-bound illustration of the traffic the hardware must move.

The relevant quantity here is arithmetic intensity, the amount of computation performed for each byte moved. Decode often has low arithmetic intensity: lots of bytes, relatively little new math, and narrow matrix operations that are harder to keep busy.

A GPU can show substantial utilization while its tensor cores are not close to their theoretical peak.

There are exceptions. Sliding-window attention reads only a bounded history. Some newer architectures store a different latent representation rather than conventional full K and V tensors. Large decode batches can make matrix operations more efficient.

Those cases change the numbers, not the underlying method: calculate what state is stored, how often it is read, and whether the workload is limited by capacity, bandwidth, or compute.

Contiguous allocation wastes the cache before you use it

The original serving pattern was simple: reserve one contiguous cache region for each request, often sized for the maximum sequence length.

That is convenient for a kernel. It is wasteful for real traffic.

Imagine a server allowing 8,192 tokens per request. A conversation that ends at 1,000 tokens uses only about 12.2 percent of its reserved region.

The remaining 87.8 percent is reserved but empty. A second request may need a different contiguous region, even if thousands of small holes exist elsewhere in memory.

Real traffic makes this worse. One request may finish after 80 tokens, another after 2,000, and another may be cancelled halfway through generation.

The allocator is left with gaps and awkward tails. Reserving for the maximum avoids moving the cache as a sequence grows, but it turns every short request into a partially empty reservation.

Under tested workloads, earlier systems could waste 60–80 percent of available cache memory through over-reservation and fragmentation. Those percentages are not a law of nature; they depend on sequence-length distributions and allocation policy.

They describe the kind of traffic pattern that makes contiguous allocation collapse.

PagedAttention, introduced in the vLLM paper, applies the operating-system idea of virtual memory to this problem. The logical cache for a sequence is divided into fixed-size blocks.

Physical blocks can live anywhere in GPU memory. A block table maps the sequence’s logical token positions to those physical locations.

The request gets a new block only when it needs one. When a request finishes, its blocks return to the free pool.

The cache no longer needs one large contiguous reservation, and a short request does not pin thousands of empty token slots.

Paging does not make live KV data smaller. The 2.5 GiB in the example is still 2.5 GiB.

It removes wasted reservations so more live requests fit.

That distinction is important. PagedAttention is a memory-management improvement, not a magical attention algorithm.

The original vLLM evaluation reported 2–4× throughput improvements over earlier systems on its tested workloads, largely because more sequences could remain active and the GPU could be fed more effectively.

You should not copy that multiplier into a capacity plan. You should copy the design principle: do not spend scarce GPU memory on empty sequence tails.

Block size is a trade-off. Small blocks reduce unused space at the end of a sequence, but require more block-table entries and more bookkeeping.

Large blocks reduce metadata and may work better with some kernels, but waste more space in partially filled tails. The right size depends on the engine and traffic.

The point is not choosing a fashionable number. It is replacing one indivisible reservation with controlled allocation.

Continuous batching keeps short requests from starving the GPU

Memory packing solves only half the problem. The other half is deciding which sequences run together.

With static batching, the server collects a group of requests and runs them as a batch. If one request needs 1,000 generated tokens and three others need 20, 200, and 40, the batch may reserve work for 4,000 sequence-token steps.

Only 1,260 of those steps produce useful output. The simple utilization ratio is 31.5 percent.

The three short requests finish early, but their slots remain tied to the batch until the long request finishes. New arrivals wait outside.

Continuous batching, also called iteration-level scheduling, makes a scheduling decision at each decode iteration. When a sequence finishes, the scheduler can place a waiting sequence into the freed slot.

The batch changes while the server is running.

This improves throughput for two reasons:

• Completed requests no longer occupy memory or scheduler slots.
• Keeping more active sequences in each iteration creates wider batched operations, which gives the GPU more work to execute together and amortizes kernel-launch overhead.

The continuous batching pattern is therefore coupled to paged KV allocation. A scheduler can admit a new request only when it can allocate enough cache blocks for that request’s prompt and possible generation.

The memory manager and scheduler are not separate optimizations. They are two parts of one production loop.

Continuous batching is not free. Admitting too much work increases queueing and can worsen p99 latency, meaning the latency below which 99 percent of requests finish.

A long prefill can also monopolize the GPU and make already-streaming responses pause. Engines use several controls to manage that conflict:

• token budgets
• chunked prefill
• priorities
• admission limits

A server that maximizes aggregate tokens per second can be a terrible chat server if users see a 400-millisecond gap between words. Throughput and responsiveness are related, but they are not the same metric.

The next gains come from sharing and shrinking state

Once cache blocks exist independently, an engine can share blocks for an identical prefix. This is prefix caching.

Suppose every request begins with the same 2,000-token system prompt and then adds a different user question. The system prompt’s computed KV state can be reused instead of being prefetched and stored separately for every request.

Prefix caching helps only when the prefix is actually identical at the token level. “These prompts mean the same thing” is not enough.

A timestamp, request identifier, changing tool list, or different whitespace can destroy the hit. Shared blocks also need careful tenant and authorization boundaries.

Never let a cache key make one customer’s private prefix visible to another customer.

Model architecture is another lever. Grouped-query attention reduces the number of KV heads while keeping more query heads.

With the same layers, head dimension, sequence length, and precision, reducing KV heads from 32 to 8 reduces the raw cache by a factor of four. That is one reason KV-head count deserves attention when choosing a model for serving.

Quantizing the KV cache can reduce both capacity pressure and memory traffic, but it is a numerical change, not just a storage setting. Validate output quality, long-context behavior, and the particular attention kernels supported by your engine.

CPU offload can extend apparent capacity, but every cache miss crossing PCIe or another interconnect adds latency. Capacity rescued by a slow transfer is not free capacity.

And sometimes the best memory optimization is to avoid an expensive request. A small model can handle classification, extraction, or routine support questions while a larger model handles difficult cases.

That is model routing, and it works alongside cache optimization rather than replacing it.

The strongest counterargument is sometimes right

The fair objection is this: modern GPUs are compute monsters, and matrix multiplication is still the central operation in a transformer. If a larger GPU increases throughput, why insist that memory is the bottleneck?

Because “LLM serving” hides several workloads.

Long prompt prefill can be compute-bound. A very large decode batch can make matrix operations compute-efficient. Tensor parallelism can shift the bottleneck to GPU-to-GPU communication.

A vision-language model may spend its time in an encoder rather than decoder attention. In any of these cases, improving a page allocator will not rescue a saturated compute pipeline.

A bigger GPU can also be the right purchase. It may provide more HBM capacity, more bandwidth, more compute, or all three.

The mistake is not buying hardware. The mistake is buying it without identifying which resource is exhausted.

Do not use the single percentage in nvidia-smi as the answer.

Record these measurements separately:

• prefill and decode timings
• active sequence count
• KV-cache usage
• generated tokens per second
• TTFT
• inter-token latency
• p95 or p99 latency

On a representative load test, compare HBM throughput with tensor-core activity using a profiler. The inference metrics lesson has the useful vocabulary here.

If tensor cores are busy and HBM traffic is modest, optimize compute, kernels, model parallelism, or prompt processing.

If tensor-core activity is low, HBM traffic is high, and the cache is near capacity, a larger model card may help, but better allocation and batching should be tested first.

What I would do on Monday morning

Start with a traffic sample, not an average. Record prompt tokens, generated tokens, concurrency, cancellations, and the distribution of sequence lengths.

“Average request is 900 tokens” hides the 8,000-token requests that exhaust the cache.

Then run three baselines with the same model and the same replay:

1. One request at a time.
2. Static batching.
3. A production serving engine with block-based or paged KV allocation and iteration-level batching, such as vLLM or SGLang.

Measure TTFT separately from decode speed. A system can improve aggregate output tokens while making first-token latency worse.

Track cache occupancy and the number of active sequences at the moment requests queue or fail.

If the first visible symptom is a CUDA out-of-memory error when a new request arrives, inspect KV capacity before touching model weights.

Limit admitted tokens, control maximum active sequences, shorten or summarize old conversation history where quality allows, and consider a model with fewer KV heads or a validated KV-cache quantization mode.

If one request is fast but throughput collapses as traffic rises, look for an application-side loop that submits one request and waits for completion before submitting the next.

Also check whether the server is using static batches. The symptom is often obvious: active sequence count stays near one while the queue grows.

If answers stream smoothly until someone submits a long document, then every other response develops large gaps, you have a prefill-versus-decode scheduling problem.

Use chunked prefill or an equivalent scheduler control, impose a prompt-token budget, and test a priority policy that protects active decode work.

If prefix caching shows no benefit, log the actual prefix hit rate. Do not guess.

Compare tokenized prefixes and look for dynamic headers, timestamps, random identifiers, or per-user text placed before the shared system prompt.

Finally, fill the GPU deliberately. Leave enough headroom for activations, temporary workspaces, allocation variance, and traffic bursts.

A configuration that reaches 99 percent cache occupancy in a quiet replay is not “efficient”; it is waiting for the 3 a.m. request that is one token too long.

The practical rule is simple:

Optimize the memory path first for decode-heavy, concurrent workloads. Optimize compute first for prefill-heavy or highly batched workloads.

That is not a contradiction. It is the difference between measuring an LLM and measuring the request pattern that actually pays your GPU bill.