Long context: windows, costs and failure modes
A practical guide to what an LLM context window contains, what long prompts cost, and why more tokens can still produce worse answers.
What you'll learn
- How a context window becomes tokens, attention work, and KV-cache memory
- Why attention compute grows quadratically while KV-cache memory grows linearly
- When long context beats retrieval, and when RAG remains the better design
- Why models miss information in the middle and how context rot develops
- How reranking, compaction, edge placement, and prefix caching reduce failures and cost
Before you start
At 3 a.m., a support assistant receives a 90,000-token conversation. The customer asks whether a refund is allowed. The answer is in the company policy pasted 40,000 tokens earlier: refunds are allowed within 30 days.
The model replies, confidently, “Refunds are allowed within 90 days.”
Nothing crashed. The request fit the advertised window. The model even quoted the policy. It just used the wrong paragraph.
This is the practical problem behind long context: giving a large language model, or LLM, a long sequence of tokens to read before it generates an answer. A larger window removes some engineering work, but it does not create perfect recall, free memory, or unlimited reasoning.
What a context window actually is
A token is a piece of text chosen by the model’s tokenizer. A short word may be one token; an unusual name may be several. The context window is the maximum number of tokens in one model sequence, usually counting both input and generated output.
If a model offers a 128,000-token window and you reserve 2,000 output tokens, the input budget is roughly 126,000 tokens. System instructions, tool definitions, conversation history, retrieved documents, images represented internally, and the new question all spend that budget.
The window is temporary working material, not a hard drive. The model does not permanently learn what you put into one request.
In a conventional decoder-only Transformer with dense, full causal attention, each layer produces a key and value vector for every token. The key helps a later token decide whether a previous token is relevant; the value carries the information retrieved when it is. These vectors remain in the KV cache while the answer is generated.
The formula below assumes dense attention. Grouped-query attention (GQA) and multi-query attention (MQA) store fewer or shared key-value heads. Latent-attention, sliding-window, and recurrent-state architectures may cache compressed or bounded state instead, so they need architecture-specific accounting.
Why attention gets expensive
In standard dense self-attention, every token compares itself with every token it can see. A sequence of length n therefore creates roughly n × n query-key interactions. Doubling a prompt from 16,000 to 32,000 tokens produces about four times as many interactions.
Memory-efficient kernels such as FlashAttention avoid materialising the entire attention matrix. They save memory, but do not make the underlying full-attention work linear.
Separate two phases:
- Prefill processes the existing prompt, so dense attention has quadratic-in-length compute.
- Decode generates one token at a time. Each new query reads the cached keys, so work for that token grows roughly linearly with context length.
The KV cache prevents the prompt’s keys and values from being recomputed for every output token. It therefore grows linearly with sequence length:
KV bytes = 2 × L × H × D × N × BHere L is the number of layers, H stored key-value heads, D head dimension, N sequence length, and B bytes per stored number. The first 2 accounts for keys and values. Batch size and concurrent requests multiply the result.
A real arithmetic example
Imagine a model with 32 layers, 32 key-value heads, head dimension 128, 32,768 tokens, and bfloat16 storage (2 bytes per number).
For one token in one layer:
2 × 32 heads × 128 values × 2 bytes = 16,384 bytesAcross 32 layers:
16,384 × 32 = 524,288 bytes per tokenAcross 32,768 tokens:
524,288 × 32,768 = 17,179,869,184 bytesThat is exactly 16 GiB, using 1 GiB as 2³⁰ bytes.
With GQA and only 8 stored KV heads, the same sequence needs 4 GiB rather than 16 GiB. Four concurrent sequences in the original configuration need about 64 GiB just for KV state, before model weights, temporary activations, or framework overhead. This is why long prompts can cause out-of-memory errors even when short prompts fit.
The practical distinction is simple: dense attention work grows steeply with length, while stored K/V state grows linearly under this architecture.
Why long context does not kill RAG
Retrieval-augmented generation, or RAG, selects relevant pieces of an external corpus and puts them into the prompt. A larger window lets you include more pieces; it does not make selection unnecessary.
At the hypothetical rates of $3 per million input tokens and $15 per million output tokens, a 100,000-token request with a 500-token answer costs $0.3075. A 20,000-token request costs $0.0675. At 10,000 requests per day, that difference is $2,400 per day. Provider prices vary, but input processing also increases time to first token.
Retrieval improves precision as well as cost. A reranker can select three passages about refund eligibility instead of making the model compete with old policies, duplicated language, examples, and unrelated text. More evidence can become more distraction.
Long context is usually the better choice when:
- one known document or small corpus fits comfortably;
- the task requires comparison across the whole artifact;
- the source is already in hand and the task is one-off;
- indexing and maintaining a retrieval system would cost more than sending the document.
RAG is usually better for large, changing, or high-volume corpora. The common production design is hybrid: retrieve, enforce authorization, rerank, deduplicate, and send a coherent evidence bundle.
The middle is a dangerous place
A model can technically attend to a token without reliably using it. The support example illustrates the lost-in-the-middle effect: recall is often better near the beginning or end of a long prompt than for equally important information buried in the middle.
This varies by model and task, but the cause is structural. Positional representations, learned attention patterns, instruction framing, and competition among many plausible passages affect which tokens receive useful attention. The beginning often contains system framing; the end is closest to the question and next-token prediction. Middle content must compete with both.
A needle-in-a-haystack benchmark places one hidden fact among irrelevant text and tests whether the model can repeat it. It can reveal an obvious context limit, but does not prove robust document reasoning. Real tasks may contain multiple relevant facts, conflicting versions, arithmetic, and multi-step synthesis.
Test position sweeps with realistic documents. Measure citation correctness, contradiction rate, multi-hop answers, refusal behaviour, and performance with incomplete evidence.
Context rot in conversations
Context rot is the gradual loss of answer quality as a conversation accumulates irrelevant, stale, or conflicting material. The model’s weights have not changed; its working set has become worse.
A long conversation may contain corrected assumptions, repeated mistakes, obsolete tool results, old instructions, and duplicate summaries. Staying under the context limit only prevents rejection. It does not prevent diluted signal or contradictions.
Compaction is the main remedy. Replace old turns with state containing the user’s goal, confirmed facts, decisions, constraints, open questions, and links or IDs for recoverable detail. Keep the raw transcript outside the prompt so it can be retrieved when needed.
Summarisation is lossy, so store important state structurally:
customer_id: C-1842
refund_window_days: 30
policy_version: 2026-04
decision: escalate
unresolved: verify purchase dateRetrieve the original policy or transcript when the detail matters. Do not treat a summary as perfect evidence.
Before retrieval, enforce tenant, ACL, date, and policy-version filters. Then rerank and deduplicate authorized candidates. Relevance ranking is not access control.
Put stable instructions and the output schema near the beginning. Put the current question, acceptance criteria, and most important evidence near the end. Use headings and one authoritative statement with a source identifier; blindly duplicating facts can create conflicting copies.
Prefix caching: cheaper repeated context
Many applications repeatedly send the same prefix: system instructions, tool schemas, safety policy, or a product catalogue. Prefix caching stores the KV state for an exact token prefix so later requests do not prefill it again.
For 100 requests with the same 50,000-token prefix and a different 500-token question:
Without caching: 100 × (50,000 + 500) = 5,050,000 input positions
With caching: 50,000 + (100 × 500) = 100,000 input positionsThat is 50.5 times fewer repeated prefix positions. The suffix still attends to the cached prefix, so the prefix does not disappear; repeated prefill work is what is reduced.
The cache occupies KV memory, and the exact-prefix requirement matters. A changing timestamp or request ID at the top can invalidate reuse. Put stable content first and variable content after it. Providers differ in cache lifetime, eligibility, and pricing.
Failure modes and their fixes
| First symptom | Likely cause | Fix |
|---|---|---|
| A correction in the middle is ignored | Lost-in-the-middle recall or conflict | Rerank, remove stale passages, and place the authoritative decision near the end |
| Time to first token and billing rise as chats grow | Full history is resent | Compact turns, keep structured state, retrieve old detail, and cache stable prefixes |
| The request returns a context-length error | Input plus reserved output exceeds the window | Count tokens, reserve output explicitly, then trim or summarise |
| The prompt fits but answers become vague or contradictory | Context rot or too many distractors | Reduce the working set and preserve source IDs, dates, and unresolved questions |
| Long or concurrent requests run out of memory | KV cache grows with length and batch size | Lower token limits or concurrency, use fewer KV heads, and tune cache management |
Long context is a capability to budget, not a trophy number. If a 20,000-token selected dossier answers better than a 120,000-token document dump, the smaller prompt is the more sophisticated system.
What to remember
- A context window is temporary token space for instructions, history, evidence, and output.
- Dense attention is roughly quadratic in prompt length; KV-cache memory is linear under the dense-attention assumption.
- Long context suits small known corpora and whole-document synthesis. RAG suits large, changing, or high-volume corpora when authorization is enforced.
- Recall is position-dependent, and needle benchmarks do not prove robust reasoning.
- Reranking, compaction, structured state, deliberate placement, and prefix caching control cost and failure.
Quick check
Practice this in an interview
All questionsThe context window is the maximum number of tokens an LLM can attend to in a single forward pass — both the input prompt and the model's own generated output count toward this limit. Its size determines how much prior text influences each prediction, sets a hard ceiling on document length and conversation history, and drives memory and compute costs that scale quadratically with sequence length under standard attention.
Cost scales with input plus output tokens; latency scales with output tokens and model size. The highest-leverage levers are: model routing (use a small model when the task is simple), prompt caching (reuse expensive prefix computation), output length control, and batching. Together these can cut spend 60–90% without quality regression.
The core toolkit is: system prompts (role and constraints), few-shot examples (format and tone anchoring), chain-of-thought (step-by-step reasoning), and output constraints (JSON schema, stop sequences). Combining these predictably closes the gap between a capable base model and a production-ready feature.
Hallucinations occur because an LLM is trained to produce plausible next tokens, not verified facts — it has no internal truth-checking mechanism, only statistical patterns. Common causes include rare or conflicting training data, overconfident decoding, and prompts that lead the model to extrapolate beyond what it learned. Mitigation strategies include retrieval-augmented generation, grounding responses to retrieved sources, lowering temperature, and calibrated refusal training.