When the KV cache doesn't fit in GPU VRAM, what are your options?

How to think about it

Short answer: I have five levers: use GPU memory more efficiently, offload inactive KV blocks to CPU RAM, compress the cache, shard it across GPUs, or evict it and recompute the prompt when the session returns. I would keep live decoding in GPU memory; SSD and network storage are cold storage, not a practical place for attention to read on every token.

Why the KV cache becomes the bottleneck

The KV cache stores the intermediate attention keys and values for tokens the model has already seen. During prompt processing, called prefill, the model computes those values for the whole input. During decode, it generates one token at a time and attends to the previous tokens.

Without a cache, the server would recompute the old keys and values for every new token. With a cache, it avoids that computation, but it must read the cached values again for each decoding step. The cache therefore grows with both context length and the number of active sessions.

A useful estimate is:

KV bytes = 2 × layers × KV heads × head dimension × bytes per number × tokens × sequences

The 2 accounts for keys and values. The number of KV heads matters, not just the model’s total attention heads. Grouped-query attention, or GQA, lets many query heads share fewer KV heads and therefore reduces cache size.

Consider a 70B-class transformer with 80 layers, 8 KV heads, a head dimension of 128, and BF16 values at 2 bytes each:

2 × 80 × 8 × 128 × 2 = 327,680 bytes per token

That is 320 KiB per token. At 32,768 tokens, one session needs about 10 GiB of KV cache. Four such sessions need about 40 GiB, before allocator metadata and other workspace are counted.

The important consequence is that a model can fit in VRAM while its intended context length and concurrency do not. Weights are mostly fixed. KV memory grows as users talk.

What I would do first

First, establish whether the problem is genuine capacity or wasted allocation.

A paged or block-based KV allocator can reduce fragmentation by storing cache blocks in non-contiguous regions of VRAM. A scheduler can also cap maximum context length, maximum generated tokens, and the number of simultaneous sequences. Those controls prevent one long conversation from reserving memory that dozens of short requests could have used.

Shared prefix caching can help when many requests begin with exactly the same tokens, such as a long system prompt. The server stores one KV copy and lets several requests reference it. The prefix must really match, including the model, adapter, tokenizer output, and relevant position settings. A nearly identical prefix is not good enough.

If the model supports sliding-window or local attention, the server may retain only the recent window rather than the entire history. That is an architectural property, not a setting that can be safely imposed on an ordinary full-attention model.

Common mistake. PagedAttention and KV offloading solve different problems. Paging packs blocks more efficiently inside a memory tier. Offloading moves blocks between GPU memory and another tier. Paging can make 10 GiB use nearly 10 GiB instead of 12 GiB because of fragmentation, but it cannot make 10 GiB fit into 5 GiB.

Option one: offload inactive sessions

For a chat service, the most useful pattern is usually a hot and warm cache.

Active sessions remain in GPU memory. When a session has been idle for a defined period, the server copies its KV blocks to pinned CPU RAM, meaning host memory reserved for efficient device transfer. When the user returns, the server promotes those blocks back to VRAM before decoding.

This works because an idle session does not need high bandwidth. A live session does.

The bandwidth difference is severe. A PCIe Gen4 x16 connection has a theoretical one-direction rate of roughly 32 GB/s before protocol and software overhead. Moving a 10 GiB cache over a sustained 25 GB/s link takes about 0.4 seconds in one direction. That may be acceptable as a resume cost. Reading that same cache from CPU memory during every generated token is a different matter. The link then becomes part of the decode loop, and latency and throughput suffer.

Some serving systems support finer-grained or layer-wise offloading, but the same trade-off remains: less VRAM usually means more transfer traffic and worse latency. I would benchmark time to first token after resume, not just average generation speed.

CPU RAM is also not free. Large pinned allocations can create host-memory pressure, and poor NUMA placement can make transfers slower on multi-socket servers. The cache manager needs ownership, limits, and eviction rules rather than an optimistic instruction to “swap when full.”

Option two: quantize the KV cache

If the cache itself is too large, reduce the number of bytes stored per key and value.

BF16 uses 2 bytes per value. FP8, an 8-bit floating-point format, uses roughly half as much storage. An INT4 representation can use roughly one quarter as much storage before accounting for scaling factors, alignment, and metadata. In the example above, the 10 GiB cache is approximately 5 GiB with one-byte values or 2.5 GiB with four-bit values, subject to implementation overhead.

The trade-off is output quality. Quantization changes the values used by every later attention operation. Errors that are invisible on a short prompt may affect retrieval from a long context, rare names, code, or exact numerical answers. Key and value quantization can also have different sensitivity.

I would validate KV quantization on the actual workload: long-context retrieval accuracy, task success, perplexity where useful, and tail latency. Quantizing model weights is a separate decision. It may free VRAM for the cache, but it does not reduce how quickly the cache grows with tokens.

Option three: use more than one GPU

The cache can be sharded across GPUs, often alongside tensor or context parallel execution. This increases aggregate memory and can let a long-context model serve more sequences.

It is not a free multiplication of capacity. The serving engine must coordinate attention across devices, and communication may occur over NVLink, a PCIe fabric, or another interconnect with very different characteristics. A topology that looks fine on a diagram can become the bottleneck at high concurrency. GQA can also limit convenient head-wise sharding because there are fewer KV heads to distribute.

I would choose this route when the model already requires multiple GPUs or when the latency target justifies the communication cost. I would not add a slow remote GPU merely to avoid designing an eviction policy.

Option four: evict and recompute

The simplest policy is to discard the KV cache for an idle session while keeping its token IDs, conversation state, model version, and adapter identity. When the user returns, the server runs prefill again and rebuilds the cache.

Recomputation trades memory for compute. In the running example, a 32,768-token session requires rebuilding the history of 32,768 tokens. A 1,000-token session needs only about 312.5 MiB of BF16 KV, so recomputing it may be simpler than managing a transfer. For a frequently resumed long session, CPU offload will often give a better resume time. For a rarely resumed session, recomputation avoids consuming host RAM and avoids cache-version problems.

The decision should use measured prefill latency, transfer bandwidth, resume frequency, and the product’s time-to-first-token target. There is no universal cutoff.

The production pattern

I would normally use three states:

Session state	Where its KV lives	Typical action
Hot	GPU VRAM	Decode immediately
Warm	CPU RAM	Promote on activity
Cold	KV evicted; token history retained	Prefill on return

The scheduler needs a GPU cache budget separate from total VRAM. It should reserve room for activations and temporary workspace, use an idle timeout or least-recently-used policy, and avoid promoting every waiting session at once. If no session can be admitted safely, queue or reject the request rather than discovering the limit through an out-of-memory crash.

A common failure mode is cache thrashing. The first symptom is usually a sharp rise in p99 time to first token after a traffic increase, accompanied by sawtooth PCIe traffic and frequent evictions. GPU utilization may even fall because the worker is busy moving cache blocks instead of decoding. The fix is usually a smaller hot set, admission control, longer residency for genuinely active sessions, or more memory—not simply a faster eviction loop.

What they’ll ask next

Why not keep every KV cache in CPU RAM?
Because decode repeatedly reads prior keys and values. CPU memory reached through a device link has much less bandwidth and higher latency than GPU memory. CPU RAM is a good warm tier for inactive sessions, not the default live-decode tier.

Does PagedAttention solve a KV cache overflow?
Not by itself. It reduces fragmentation and supports efficient block allocation, but it does not reduce the logical number of KV bytes. You still need shorter contexts, fewer sessions, compression, offloading, sharding, or eviction.

How do you choose between offloading and recomputation?
Compare the time to transfer the cache with the time to prefill the retained token history. Offload when sessions are long and frequently resumed. Recompute when sessions are short or rarely revisited, or when host-memory pressure is more costly than extra GPU compute.

Say this in the interview: “I would keep active KV on GPU, page and schedule it carefully, offload idle blocks to CPU RAM, quantize or shard when capacity is still short, and evict plus recompute cold sessions; SSD is for durable history, not live attention.”