When the KV cache doesn't fit in GPU VRAM, what are your options?
The KV cache is working memory — it's re-read to generate every token — so it has to stay fast. When VRAM fills, you offload the least-active sessions down a memory hierarchy: GPU VRAM (active, ~3 TB/s), CPU RAM over PCIe (idle, ~50 GB/s), local SSD (long-idle), and networked storage (cold/durable only, never live decode). Idle sessions are parked lower and promoted back to VRAM on activity. The alternative is to drop the cache and recompute the prefill when the session returns; for long prompts, offloading and reloading usually beats recomputing attention over thousands of tokens.
How to think about it
The key framing is that the KV cache is working memory, not stored data: during generation the model re-reads the entire cache for every token, so the tier it lives in directly bounds decode latency.
The hierarchy (bandwidth vs capacity):
| Tier | Bandwidth | Use |
|---|---|---|
| GPU VRAM | ~3 TB/s | active sessions — the only place fast enough for live decode |
| CPU RAM (PCIe) | ~50 GB/s | idle sessions; promote back to VRAM on activity |
| Local SSD | ~5 GB/s | long-idle sessions |
| Networked SSD | ~1 GB/s | durable storage / recovery only — never live decode |
Two strategies when VRAM is full:
- Offload the cache down a tier and reload it when the session resumes.
- Evict + recompute the prefill on return.
For long contexts, reloading from CPU RAM is cheaper than re-running attention over the whole prompt, so offloading wins; for short prompts, recompute is fine.