A vendor claims its model supports a 200k-token context, but your evaluation shows that it misses evidence placed in the middle of long documents. How would you diagnose the failure, and when would you use retrieval or summarization instead of simply increasing the context window?

How to think about it

I would first separate the vendor’s maximum context capacity from its effective context quality: measure answer accuracy as evidence moves through the document, while checking for truncation and formatting bugs. I would use retrieval for sparse, citable facts, summarization for broad but lossy compression, and a larger context only when the task genuinely needs global relationships and testing shows the model can use them reliably.

What the 200k-token claim does and does not mean

A context window is the amount of tokenized input the model can accept in one request. It is a capacity limit, not a guarantee that every token receives equal attention.

The likely failure is called lost in the middle. Models often use information near the beginning or end of a long prompt more reliably than equally important information buried halfway through it. That pattern is not magic, and it is not necessarily a broken tokenizer. A transformer computes relationships between tokens through attention, but attention is a competition: as the sequence gets longer, many irrelevant tokens compete with the few tokens that answer the question.

The model must also do more than locate a sentence. It must notice it, connect it to the question, resolve conflicts with nearby evidence, and reproduce the answer. Each step can fail. Long documents contain repeated names, boilerplate, contradictory dates, tables, definitions, and instructions. A single useful sentence can be statistically quieter than 100 pages of legal boilerplate.

Training matters too. A model may technically accept 200k tokens while having seen fewer examples that require reliable reasoning over the middle of a 150k-token sequence. Positional encoding and long-context training can affect quality at different distances, although the exact behavior is vendor- and model-specific.

There is a more embarrassing possibility: the model never received the evidence.

A claimed 200k-token limit may refer to input alone, or to the combined input and output budget. An SDK, gateway, or application may reserve output space, truncate old messages, drop a document chunk, mishandle PDF text, or count tokens differently from your local tokenizer. Diagnose those before blaming attention. Otherwise you can spend a month tuning retrieval to compensate for a missing page.

How I would diagnose it

I would build a controlled needle-in-a-haystack test: hide a known fact, the needle, inside a long document made mostly of realistic distractor text, then ask a question whose answer requires that fact.

Do not run one impressive demo. Run a matrix.

1. Keep the document, question, model version, decoding settings, and expected answer fixed.
2. Insert the same fact at several normalized positions: near the start, one-quarter in, the middle, three-quarters in, and near the end.
3. Randomize the exact location between trials so the model cannot learn a fixed position.
4. Repeat each condition enough times to report uncertainty, not just one success or failure.
5. Test several total lengths, such as 32k, 64k, 128k, and 200k tokens.
6. Test one needle, multiple needles, paraphrased questions, tables, code, and conflicting distractors.

I would score more than exact final-answer accuracy. Ask the model to quote the supporting sentence and provide its source offset or page. Measure:

• whether it found the correct evidence;
• whether the answer is correct;
• whether the citation actually supports the answer;
• abstention when the evidence is absent;
• latency, input tokens, output tokens, and cost.

The first production symptom of this failure is often a polished wrong answer: the model confidently cites a nearby section, but the relevant clause is somewhere else. A citation that sounds plausible is not evidence that the model read the right passage.

Here is a concrete synthetic test for a 120k-token vendor contract. The question is: “What is the provider’s liability cap for a security incident?” The exact answer appears once, at roughly token 60,000. Other sections contain different caps for service credits, ordinary negligence, and intellectual-property claims.

Suppose the result looks like this:

Evidence position	Correct answer rate
First 5 percent	19 of 20
Middle 50 percent	7 of 20
Last 5 percent	18 of 20

That is a position-sensitive failure, not proof that the model cannot answer the question. I would then repeat the test with the same contract reduced to 8k tokens. If performance returns to 20 of 20, the long-context behavior is the leading suspect.

Before concluding that, I would inspect the request sent over the wire. I would verify:

• the final serialized prompt contains the needle exactly once;
• the document is not truncated before inference;
• the vendor’s reported token count matches the request’s actual structure;
• system, developer, user, and document content are in the intended order;
• PDF extraction has not removed table cells or split a sentence;
• the application has not discarded chunks while assembling the prompt;
• the model and endpoint are the tested versions;
• the output reservation has not reduced the usable input budget.

I would also run a short-context control, a direct quote task, and a source-only task. If the model can quote the middle sentence when shown only the relevant page but misses it in the full document, that strongly suggests long-context retrieval or distraction. If it cannot quote the sentence even in isolation, the problem may be extraction, instruction following, or the evaluation itself.

When I would use retrieval

Retrieval means selecting a smaller set of relevant passages from a larger corpus before asking the model to answer. It changes the problem from “search 120k tokens while reasoning” to “reason over the best 10k tokens.”

I would choose retrieval when:

• the answer usually depends on a small fraction of the corpus;
• the source changes frequently;
• answers need page-level citations or auditability;
• many users ask different questions over the same document set;
• the corpus is larger than one practical context window;
• latency and token cost matter.

For the contract example, I would combine lexical search for exact terms such as “security incident” and “liability cap” with semantic search for paraphrases. I would rerank the candidates, include neighboring paragraphs, and pass source IDs and page numbers to the model. Retrieval and long context are not opposites: a 200k-capable model can still be useful as the reasoning engine over retrieved evidence.

Retrieval has its own failure mode. The first symptom is often a confident “the document does not specify this” answer, even though the clause exists. Causes include poor chunk boundaries, an overly broad query, missing synonyms, OCR errors, or a retriever that prefers semantically similar but legally irrelevant text. I would therefore measure retrieval recall separately: did the correct passage enter the candidate set before the model saw it?

When I would use summarization

Summarization compresses a document into a shorter representation. It is useful when the task needs the shape of the whole document: the main themes, a timeline, duplicate recommendations, or a comparison across 80 sections.

For a 120k-token board-meeting archive, I might summarize each meeting, then summarize those summaries, while preserving dates, decisions, owners, and links to original passages. That is hierarchical summarization. It keeps the working context manageable and makes repeated questions cheaper.

But summarization is lossy. It can omit the one exception that controls the legal answer, normalize two conflicting numbers into one, or repeat an early mistake through every later summary. I would not rely on a generic summary to answer “What exact termination notice period applies to enterprise customers?” For that, retrieve the original clause and cite it.

A strong production design often combines both: retrieve likely evidence, use summaries to orient the model across the whole corpus, then answer from original passages. The summary is a map. It is not the deed.

The senior-level trade-off

Increasing the context window can be the correct answer when evidence is densely interconnected. A research question may require comparing definitions from the beginning, methodology in the middle, and limitations at the end. Retrieval can damage that task by separating passages whose meaning depends on their broader context.

The mistake is treating a larger window as a universal fix. More tokens generally mean more input cost and often more latency; if pricing is linear, doubling input tokens roughly doubles input-token spend. More importantly, adding irrelevant text can reduce accuracy. A 400k-token window with the same position bias and more distractors may be a larger filing cabinet, not a better search system.

I would choose based on measured effective evidence recall, not the advertised maximum:

• use long context when global interactions matter and position tests are strong;
• use retrieval when evidence is sparse, changing, or needs citations;
• use summarization when broad coverage matters more than exact wording;
• use a hybrid when the task needs both orientation and precise proof.

The benchmark should include the failure cases your users care about, not just average accuracy on short prompts.

What they’ll ask next

“Would lowering temperature fix lost in the middle?”
Usually not. Lower temperature can reduce sampling variation, making the same failure more repeatable, but it does not repair the model’s ability to locate evidence. I would use deterministic settings for diagnosis, then fix prompt construction or information selection.

“How do you know whether retrieval or the model is at fault?”
Evaluate the pipeline in stages. First ask whether the retriever returns the correct passage. Then test whether the model answers correctly when given that passage alone. A failure in the first stage is retrieval; a failure in the second is extraction, reasoning, or answer-generation quality.

“What would you monitor in production?”
I would log source IDs, retrieved ranks, document positions, token counts, truncation events, citation validity, abstentions, latency, and cost. I would periodically replay a position-balanced evaluation set so a model update cannot quietly make middle-of-document failures worse.

“A 200k-token window tells me what the model can ingest; a position-balanced evaluation tells me what it can actually use.”