KV cache offloading & the memory hierarchy
When the KV cache outgrows GPU VRAM, tier it across VRAM, CPU RAM, and SSD. Why the cache is working memory, and which tier to never use for live decode.
What you'll learn
- Why the KV cache behaves like working memory, not stored data
- The four-tier hierarchy — GPU VRAM, CPU RAM, local SSD, networked SSD
- How offloading trades bandwidth for capacity, and what that costs per token
- When to offload, and which tier to never put live decode on
Before you start
KV caching is what stops a model re-reading the whole prompt for every new token — but that cache has to live somewhere, and at scale it won’t all fit on the GPU. This lesson is about what happens then: the memory hierarchy, and how to tier the cache across it.
The KV cache is working memory
A subtle but crucial point: the KV cache isn’t archived data you occasionally look up. During generation the model re-reads the entire cache to produce every single token. It behaves like working memory — constantly, hotly accessed. That’s why where it sits decides your latency, not just whether it fits.
Run the numbers and the pressure is obvious. A 7B model with vLLM might allocate ~5,625 cache blocks of 16 tokens — room for roughly 90k tokens of KV. Fine for one user on an L4. But serve many users, or one user with a very long context, and that budget evaporates fast.
Four tiers, bandwidth vs capacity
When VRAM fills, you don’t drop sessions — you offload the least-active KV cache down a hierarchy, each tier bigger and slower than the last. Drag the demand below and watch the cache fill the fast tier, then spill:
The KV cache is working memory: it's re-read for every generated token, so where it sits sets your latency. Offloading trades speed for capacity — promote a session back to VRAM the moment it's active again.
- G1 — GPU VRAM (~3 TB/s): the only place fast enough for active decode. Limited to tens of GB. Active sessions belong here.
- G2 — CPU RAM (~50 GB/s over PCIe): much larger, ~60× slower. Good for sessions that have gone idle — park them here, promote back to VRAM when the user sends the next message.
- G3 — local SSD (~5 GB/s): huge capacity for long-idle sessions; slow enough that you’d never decode directly from it.
- G4 — networked SSD (~1 GB/s): for durable storage — logs, checkpoints, recovering a session after a restart — not live inference. The latency would stall generation.
Offload vs recompute
There’s an alternative to offloading: just drop the cache and recompute the prefill when the session returns. Which wins depends on the prompt. Re-loading a long context from CPU RAM is often cheaper than re-running attention over thousands of prompt tokens — so offloading wins for long prompts. For short prompts, recompute can be simpler and just as fast. Production servers (vLLM, SGLang, TGI) implement both and choose based on length and pressure.
Check yourself
Quick check
What to remember
- The KV cache is working memory — re-read for every token — so the tier it sits in sets your decode latency.
- The hierarchy trades bandwidth for capacity: G1 VRAM (active) → G2 CPU RAM (idle) → G3 local SSD (long-idle) → G4 network (cold storage only).
- Offload idle sessions, promote on activity; offloading usually beats evicting + recomputing prefill for long prompts.
- Offloading (across tiers) is complementary to PagedAttention (packing within a tier) — real servers do both.
Questions about this lesson
Why can't the KV cache just stay in GPU memory?
It can for a single user with moderate context, but the KV cache grows with both the number of concurrent sessions and the length of each context, and GPU VRAM is only tens of GB. Serve many users or very long contexts and the cache exceeds VRAM. Rather than dropping sessions, servers offload the least-active KV cache to larger, slower tiers (CPU RAM, then SSD) and promote it back to VRAM when the session becomes active again.
Is KV cache offloading the same as PagedAttention?
No. PagedAttention fights fragmentation within a single tier — it stores KV in non-contiguous fixed-size blocks so GPU VRAM packs tightly and little is wasted. Offloading addresses running out of the tier entirely — it moves cache across the VRAM → CPU RAM → SSD hierarchy. They are complementary, and production servers like vLLM do both: paged blocks inside each tier, tiered placement across them.
Offload the KV cache or recompute it — which is better?
It depends on prompt length. Offloading preserves the cache and reloads it over PCIe when the session resumes; eviction drops it and re-runs the prefill (attention over the whole prompt) on return. For long contexts, reloading from CPU RAM is much cheaper than recomputing thousands of tokens, so offloading wins. For short prompts, recompute is simple and about as fast. Mature servers implement both and choose based on length and memory pressure.
Practice this in an interview
All questionsThe KV cache is working memory — it's re-read to generate every token — so it has to stay fast. When VRAM fills, you offload the least-active sessions down a memory hierarchy: GPU VRAM (active, ~3 TB/s), CPU RAM over PCIe (idle, ~50 GB/s), local SSD (long-idle), and networked storage (cold/durable only, never live decode). Idle sessions are parked lower and promoted back to VRAM on activity. The alternative is to drop the cache and recompute the prefill when the session returns; for long prompts, offloading and reloading usually beats recomputing attention over thousands of tokens.
During autoregressive generation, attention recomputes Keys and Values for all previous tokens at every step; the KV cache stores those K and V tensors so each new token only computes its own, turning per-step cost from quadratic to linear in sequence length. The tradeoff is memory growth proportional to sequence length and batch size.
The KV cache stores the key and value tensors computed during previous forward passes so they do not need to be recomputed for every new token during autoregressive generation. Without it, generating each token would require a full forward pass over the entire context from scratch, making inference cost grow quadratically with sequence length rather than linearly.
PagedAttention stores the KV cache in non-contiguous fixed-size blocks like OS virtual-memory pages, eliminating the fragmentation and over-reservation of contiguous KV allocation and enabling sharing across sequences. Continuous batching dynamically adds and removes requests from a batch at the token level instead of waiting for the whole batch to finish, sharply improving GPU utilization and throughput.