Context engineering: why accumulated context can degrade your agent

At 3:07 a.m., a support agent is handling order #4817. The customer says the package arrived damaged and wants a refund.

The agent finds the order, checks the payment, reads the refund policy, asks for a photograph, checks the photograph, and calls the refund system. At first it behaves perfectly.

Then, after a dozen or so tool calls, it asks for the order number again. A few turns later it says the refund has already been issued. Then it tries to issue it a second time.

The model did not suddenly become less intelligent. We gave it a transcript-shaped junk drawer and asked it to think clearly inside it.

The usual response is to rewrite the system prompt. Add “be concise.” Add “carefully review the conversation.” Add a stern paragraph in capitals.

This can help around the edges, but it does not repair a badly managed working state.

For multi-step agents, context engineering complements prompt engineering. Prompt wording still matters. It defines the rules.

But context engineering decides which facts, tool results, instructions, and memories reach the model on each turn. That is where long-running agents usually go wrong.

The prompt is only one layer

Prompt engineering means designing the instructions given to a language model:

• its role
• rules
• examples
• tone
• constraints

It is mostly about wording a relatively stable instruction layer.

An agent has a different problem. Its input changes after every action.

The model may receive:

• a system message
• a user request
• tool definitions
• a conversation transcript
• retrieved documents
• previous decisions
• error messages
• the latest tool result

People often call all of that “the prompt,” but treating it as one undifferentiated blob is the beginning of the trouble.

For order #4817, imagine a plausible request assembled after several steps:

• 1,200 tokens of system and policy instructions
• 350 tokens from the customer
• 2,000 tokens describing available tools
• 12 tool responses averaging 3,500 tokens each
• a few thousand tokens of previous dialogue and model reasoning summaries

That is already around 46,000 input tokens.

Most of the tool responses are not useful anymore. They contain repeated customer fields, verbose JSON, audit metadata, product catalog entries, and payment records that were relevant for one decision three minutes ago.

The model still sees them. It has no natural “that part is finished” marker unless we create one.

A better system prompt cannot make obsolete payment metadata become relevant again. It can only compete with it for the model’s attention.

Context is working memory, not a database

A token is a small piece of text that a model processes. It may be a whole short word, part of a longer word, punctuation, or whitespace.

A context window is the maximum amount of tokenized input and output the model can handle in one request.

The window is not a database. It is closer to working memory.

On every model call, the application assembles a new request. In a simple chat implementation, that means resending the system instructions, the conversation so far, and the latest tool results.

Some platforms store the transcript for you, but the model still has to process the usable context for the call. There is no magical filing cabinet in which every old fact remains equally available.

The model uses attention to relate pieces of the input to one another. In standard full self-attention, every position can interact with many other positions, so the amount of possible interaction grows roughly with the square of sequence length.

Optimizations can reduce the computational cost, and newer models can handle much longer inputs. Neither fact turns a 100,000-token transcript into a well-indexed knowledge base.

Longer context creates several kinds of interference:

• Relevant facts compete with irrelevant facts.
• Old decisions sit beside newer corrections.
• Similar entities acquire ambiguous references. “The refund” might mean a proposed refund or a completed refund.
• Retrieved documents can contain instructions aimed at changing the agent’s behavior.
• The important fact may be buried in the middle of repetitive material.
• The model has to infer what is current instead of being told explicitly.

That third-party instruction problem is indirect prompt injection, where untrusted content contains instructions designed to alter the model’s behavior.

Treat the following as untrusted data:

• customer text
• retrieved documents
• tool output

Keep trusted policy instructions separate from that data. The tool gateway should not let untrusted content authorize a tool call, and deterministic policy code should validate every consequential action.

Delimiters and labels can make the boundary clearer to a model, but they are not security controls. Compaction and sub-agent summaries can preserve or even amplify a malicious instruction, so carry source IDs and trust labels through those transformations.

This is context rot: as the context grows, the model’s ability to use the information accurately can degrade, even while the request remains below the advertised token limit. Anthropic uses the term for this broader loss of usefulness as context accumulates.

It is not a single universal threshold. A 40,000-token context can be easy for one task and poisonous for another.

A short transcript full of contradictory state may be worse than a long, carefully selected policy document.

That distinction matters when evaluating a model. If an agent performs well for five turns and fails at turn twenty, the problem may not be the model’s reasoning ability.

The assembled context may have crossed a quality boundary long before it crossed a size boundary.

The smallest useful working set

“Smallest” does not mean “shortest at any cost.” It means the smallest set that contains everything required for the current decision.

For order #4817, the next refund decision may need the order ID, payment status, damage evidence, customer identity, refund policy version, and whether a refund has already been issued.

It does not need the raw response from the product-catalog search that happened ten turns earlier.

This is the central change in mindset. Do not ask, “What can we fit in the window?”

Ask, “What must the model see to make this decision correctly?”

Four techniques make that practical.

Compaction turns a transcript into state

Compaction is the deliberate replacement of older raw conversation and tool output with a shorter summary. It is not merely truncating the beginning of a transcript.

Truncation throws away history without deciding what matters. Good compaction preserves the state needed to continue.

A related but different operation is tool-result clearing, which removes selected old tool-result content from future requests without producing a semantic summary of what it meant.

Context editing is the broader runtime capability for changing the context assembled for a model call. An implementation may use it to clear old tool results, trigger summarization, or apply other editing rules.

Compaction and summarization create new state and can lose meaning. Tool-result clearing removes content and can lose evidence. Do not treat their measurements as interchangeable.

A useful summary for #4817 might look like this:

Case: 4817
Goal: Decide whether to issue a refund for damage in transit.
Verified facts: Order delivered on 2026-08-27; payment captured; customer identity verified.
Evidence: Customer photo attached; inspection result says packaging damage is consistent with transit damage.
Policy: Refund allowed under policy version 2026-07, provided no prior refund exists.
Decision: No refund has been issued.
Open question: Confirm refund amount from the payment record.
Next action: Read the current payment status, then request a refund through the server-side refund operation.
Sources: order-result-8841, inspection-result-1192, policy-2026-07.

That last line is a plan, not an at-most-once guarantee. The model and its summary are advisory.

The refund service must enforce the financial safety rule:

• use an idempotency key for the intended refund
• perform an atomic conditional state transition so only one request can move the refund from an eligible state to a pending or completed state
• perform a fresh authorization check against current payment and order state
• enforce a refund cap or human approval for high-risk amounts

Those controls handle retries, races, stale reads, and duplicate tool calls. Context can tell the model what to try; it cannot make the transaction safe.

Notice what is preserved in the summary:

• decisions
• facts
• uncertainty
• source references
• the next action

The summary does not pretend that every discarded sentence was equally valuable.

Compaction should happen before the agent is already failing. Leave room for the next response and its tool results.

A threshold based on a fraction of the supported window is a useful starting heuristic, but it is not a law. The correct threshold depends on the task, the model, the expected tool payload, and the amount of output you reserve.

Anthropic’s public context-engineering report describes an internal evaluation using Claude Sonnet 4 on a 100-turn long-horizon tool-use workflow.

The baseline was the same workflow with context editing disabled. Anthropic reports an 84 percent reduction in token consumption across the run, with the unedited run as the denominator: (baseline token consumption - edited token consumption) / baseline token consumption.

The edited run completed the 100 turns while the baseline failed from context exhaustion.

The public report does not publish a more specific task description, raw token totals, number of independent runs, confidence interval, or whether “token consumption” includes output tokens.

Treat the result as directional evidence about that setup. It is not a benchmark showing that compaction alone saves 84 percent on every workload.

Compaction has a cost. It is lossy compression. If the agent needs the exact wording of a contract clause, a 100-token summary may be unsafe.

Keep the original evidence outside the active context and retain a pointer that can retrieve it again.

Isolation gives difficult work a clean room

Sub-agent isolation means sending a bounded sub-task to a separate model call with its own context, then returning only a distilled result to the main agent.

Suppose the support agent needs to determine whether the damage qualifies under a 40-page returns policy. The main agent does not need the entire policy, the search transcript, and every paragraph the policy-reading model considered.

A worker can receive the policy and the case evidence, then return:

Eligible: yes.
Reason: Transit damage is covered within 30 days when delivery evidence is present.
Exception: No refund if a prior replacement was accepted.
Source: returns-policy section 4.2.
Confidence: high.

A clean context reduces interference. The policy worker can focus on policy interpretation while the main agent focuses on the transaction.

The boundary must be explicit. A sub-agent that returns “looks fine” has hidden the very evidence the parent may need.

Ask for:

• conclusions
• supporting source IDs
• uncertainty
• unresolved exceptions

Returning roughly 1,000 to 2,000 tokens is often more useful than returning an entire transcript, but the right size is determined by the sub-task.

Isolation is not automatically better. It adds another model call, another failure boundary, and often more latency.

Do not isolate a tiny task that needs the main agent’s immediate conversational nuance. Use it when a sub-task has a different information diet: policy research, codebase exploration, document comparison, or a large search.

Just-in-time retrieval beats preloading

Just-in-time retrieval means keeping small identifiers in context and fetching detailed data only when the current step requires it.

The main agent might carry:

order_id: 4817
payment_id: p_991
policy_version: 2026-07

It can then request the current payment status when it is ready to decide. It does not need the entire order record, every payment event, and the full policy document from the beginning.

This improves more than token count. It reduces staleness. A payment status fetched at the moment of authorization is more trustworthy than a payment status copied into a summary several turns earlier.

Retrieval must still be engineered. A search tool that returns 200 loosely related documents has simply moved the junk drawer into a tool response.

Use:

• narrow queries
• pagination
• field selection
• relevance limits
• a compact result format

Return the five useful records, not the warehouse inventory and the database query plan.

There is also a permission issue. Just-in-time access should not mean unrestricted access.

The tool should fetch only the records the agent is allowed to see, for the purpose it is allowed to perform. Context reduction is not a reason to weaken data governance.

Code execution keeps bulk data out of the window

Sometimes the right answer is not to show the model the data at all.

Suppose an operations agent must inspect 50,000 support tickets for duplicate refund requests. Pasting all ticket text into the context is expensive and gives the model an enormous retrieval problem.

A sandbox can read the records, filter by order ID, group by customer, calculate counts, and write a small result file. The model sees the aggregate, the suspicious cases, and perhaps a few selected examples.

This pattern becomes especially useful with MCP, the Model Context Protocol, which standardizes how a host exposes tools and resources to a model.

MCP does not itself provide code execution, sandboxing, or a way to keep intermediate results out of the model-visible conversation.

A separate host-side code-execution adapter can invoke MCP tools inside a controlled sandbox, retain intermediate data there, and return only selected aggregates or files to the model.

The data stays out of the conversation because of that adapter and sandbox boundary, not because of MCP alone.

Anthropic’s code-execution-with-MCP report describes one workflow in which an ordinary MCP integration carried roughly 150,000 model-visible tokens through the conversation, while a filesystem-based API processed through code execution exposed about 2,000 tokens to the model.

Using the ordinary workflow as the denominator, (150,000 - 2,000) / 150,000 is about 98.7 percent.

The source presents this as one workflow with rounded token counts, not as an average across a disclosed sample. The saving came from changing where the filtering happened.

The model stopped carrying the intermediate rows through its conversation.

This is powerful, but it moves risk rather than abolishing it. A sandbox needs:

• restricted credentials
• network controls
• file permissions
• timeouts
• resource limits
• an audit trail

Never give an agent broad production access merely because the data no longer appears in the prompt. The code-execution-with-MCP lesson goes deeper into that boundary.

The production pattern is an external state loop

The reliable architecture is not “append another message forever.” It is an external state loop.

Store durable state outside the transcript. That state should contain:

• facts
• decisions
• identifiers
• permissions
• open questions
• references to raw evidence

The transcript becomes an event log for debugging, not the only source of truth.

On each turn, assemble context from five pieces:

1. Stable instructions and safety rules.
2. The current user goal.
3. The small structured state relevant to that goal.
4. Only the tools needed for the next action.
5. Fresh evidence retrieved for the current decision.

After a tool call, normalize the result before showing it to the model. Strip duplicate fields, verbose metadata, irrelevant records, and completed work.

Store the raw response externally with an ID so it remains auditable and retrievable.

Your structured state should have one authoritative answer for important facts. For money movement, however, even that state is advisory.

The refund service’s current database state and authorization code—not the transcript, summary, or model judgment—must be the source of truth.

If the transcript says a refund is pending, a stale tool result says it failed, and a summary says it succeeded, the model is being asked to resolve a database conflict through prose.

That is a poor database design disguised as an AI problem.

A durable memory system helps with facts that should survive a session, but memory is not a licence to load everything into every request.

The useful principles in agent memory design are the same ones used here: decide what deserves persistence, attach provenance, and retrieve by need.

The strongest objection is reasonable

The strongest counterargument is that context windows are getting much larger and models are getting better at using them.

Why build a complicated compaction and retrieval layer when a model can accept hundreds of thousands of tokens?

For some workloads, you should not build one. A single document review, a short-lived coding task, or an investigation where every paragraph is relevant may benefit from broad context.

Summarizing too aggressively can destroy exact wording, legal exceptions, or a subtle relationship between two pieces of evidence.

The mistake is treating context engineering as a command to minimize tokens blindly. It is a command to maximize useful signal.

A larger window is valuable headroom. It lets you include more relevant evidence and reduces the chance of hard overflow.

It does not remove input cost, latency, stale state, tool-output noise, or ambiguity. Nor does it prove that the model used every fact correctly.

Measure the actual task. If full context improves accuracy without unacceptable cost or latency, keep it.

If performance falls as irrelevant history accumulates, curating the context is not premature optimization. It is the fix.

What to change on Monday morning

Instrument and test the run

Start by instrumenting the assembled request, not just the final answer.

For every model call, record:

• the input token count
• the output token count
• tool-result sizes
• message or state version
• retrieval IDs
• compaction events
• latency
• retries
• final task outcome

Redact personal and secret data before storing traces. The point is to see whether failures arrive with a growing context, repeated tool payloads, or contradictory state.

Agent observability is useful here because the important unit is the whole run, not one model response.

Next, build one long fixture around order #4817 or an equivalent real workflow. Run it with five turns, twenty turns, and fifty turns.

Put the critical fact near the beginning, middle, and end. Add irrelevant tool fields and one deliberate correction.

Test whether the agent:

• chooses the right tool
• preserves the refund decision
• lets the refund service reject a duplicate request even if the agent retries

A good short-context answer proves very little about a long-running agent.

Shape and validate the state

Then separate raw evidence from active state. Store complete tool responses outside the model context.

Replace each response with a compact, typed result containing the fields required for the next decision and a source ID.

Do not let every tool decide its own prose format. Inconsistent verbose output is how token budgets disappear.

Add a summary contract before adding clever summarization. Require fields for:

• the goal
• verified facts
• decisions
• open questions
• next action
• uncertainty
• source references

Represent “unknown” explicitly. An omitted field can mean unknown, false, or forgotten; those are not interchangeable in a refund workflow.

Finally, compact before the hard limit and reserve space for the next tool call. Try an initial guardrail around 60 to 70 percent of the advertised input capacity, then tune it from traces.

Keep the last few turns and the structured state available after compaction. If a summary fails validation, fall back to the previous checkpoint rather than letting a fluent but incomplete summary become the new truth.

Diagnose failures as state problems

The first failure you will probably see is not an exception. It is a plausible answer with the wrong state: “The refund was already processed,” followed by a duplicate action.

That symptom points to stale or contradictory context. Look for two competing payment statuses before touching the wording of the prompt.

Then verify that the refund service—not the context—enforces:

• the idempotency key
• the fresh authorization check
• the atomic conditional transition
• the refund cap or approval rule

A second failure looks like retrieval thrashing. The agent calls search repeatedly, receives large results, and makes no progress.

That usually means just-in-time retrieval was added without a narrow query, result limit, or stopping condition. Reduce the payload and make the next required fact explicit.

A third failure is a beautifully compact summary that has quietly dropped an exception. This is the danger of optimizing for token count instead of decision quality.

Preserve source references, test summaries against exact cases, and retrieve the original evidence whenever the decision is high-stakes.

The best agent is not the one that can carry its entire past. It is the one that knows what its present decision requires, keeps that state clean, and leaves the rest somewhere it can be fetched without cluttering the room.