Context engineering

At 3:07 a.m., a support agent is handling a refund. It has read the order record, queried the payment provider, searched the returns policy, inspected two earlier tickets, and tried an API call that timed out. The customer has sent six more messages.

The agent still has the answer somewhere in its history. It also has 18,000 tokens of JSON, duplicated policy text, stale attempts, and a payment error buried between them. It now asks the customer for the order number they supplied ten minutes ago.

The problem is not that the model needs a more eloquent instruction. The problem is that the model is being shown too much.

Context engineering is the discipline of deciding what an agent sees, when it sees it, and what stays outside the model’s context window. A context window is the maximum token budget a model can process for one request, including the assembled input and, depending on the provider, room reserved for its output. A token is a small piece of text used for billing and model limits; it is not exactly a word.

Prompt engineering chooses words. Context engineering chooses the evidence, history, tool output, instructions, and state those words are surrounded by. For a one-shot question, wording may dominate. For an agent making 40 tool calls, context selection usually dominates.

The window fills before you expect it

Take a modest agent:

• system instructions: 1,500 tokens
• tool definitions: 2,500 tokens
• returns policy and output rules: 1,000 tokens
• each turn, including the model message and tool result: 1,500 tokens

The fixed part is 5,000 tokens. After 12 turns:

5,000 + (12 × 1,500) = 23,000 tokens.

On a model with a 32,000-token context window, this sounds comfortable. It is not. The application may reserve 4,000 tokens for the next answer, leaving 28,000 for input. A single 6,000-token tool response can consume most of the margin. Real responses can be much larger: a database query, browser page, or log file adds a cliff rather than a gentle slope. Treat the published limit as a ceiling, not a target.

Two failures follow.

Overflow is the obvious one. The request is rejected, the framework truncates old messages, or the application fails while assembling the prompt. Silent truncation is particularly dangerous because the agent appears healthy while losing the instruction or fact it needed.

Context rot happens earlier. A crowded context can produce worse answers even when it still fits. Relevant information competes with irrelevant information, and facts in the middle of a long sequence can be harder to retrieve than facts at the beginning or end. This is commonly called the lost-in-the-middle effect.

The model has not developed a literal memory leak. “Rot” describes declining usefulness as the context becomes noisy, repetitive, and difficult to search.

The shape is easier to see than the arithmetic. Unmanaged history rises toward the limit. A managed history repeatedly drops old detail into a smaller summary and continues in a safer band.

Three ways to keep the window useful

The practical toolkit has three moves:

1. Compaction replaces old conversation with a smaller, useful summary.
2. Isolation gives a sub-task a separate context so its working noise does not enter the parent.
3. Just-in-time retrieval fetches evidence only when the current task needs it.

Compaction manages a conversation that must continue. Isolation contains a task that can be handed off. Retrieval keeps a large external knowledge store out of the window until there is a reason to consult it.

Compaction: preserve the state, not the transcript

Compaction summarizes older turns, removes their raw messages, and keeps the summary alongside recent work. It is not gzip-style compression: the model cannot later reconstruct a deleted tool response byte for byte.

Return to the refund agent. Suppose the application compacts at 24,000 input tokens. At turn 13, it takes the first eight turns, about 12,000 tokens, and asks a model for a 1,200-token summary. It retains the five most recent turns, about 7,500 tokens, plus the 5,000-token fixed prefix:

5,000 + 1,200 + 7,500 = 13,700 tokens.

The agent drops nearly 10,000 tokens while keeping the current exchange and a record of the earlier investigation. After several more turns, the application repeats the process.

A useful summary records what the next turn can act on:

• the user’s goal and constraints
• decisions, verified facts, and their sources
• failed attempts and unresolved questions
• side effects already performed
• the next safe action

“Investigated refund” is poor. “Order 4817 was delivered on 12 August; policy allows a refund within 30 days; payment reversal has not been attempted; customer wants the original card; the payment API timed out twice” preserves state.

Keep critical state in structured fields outside the prose summary when possible. The order ID, authorization status, refund status, and idempotency key should not depend on a model remembering a sentence. The transcript can be compacted; the transaction record belongs in a database or durable state store.

Compaction has a sharp limitation: a bad summary is lossy compression with authority. If it changes “refund not issued” to “refund issued,” later turns may act on the mistake confidently. Preserve raw history in an audit store when it matters, and instruct the summarizer to mark uncertainty rather than fill gaps.

Isolation: keep a worker’s mess out of the parent

Isolation gives a sub-task its own context window. A parent agent sends a worker a narrow assignment, the worker performs the noisy investigation, and the parent receives a result rather than the worker’s transcript.

This works for tasks with a natural boundary: researching documents and returning cited findings, inspecting code and returning defects, or executing a batch and returning counts plus failures. The worker should return a typed result with status, key facts, evidence references, side effects, and a recommended next action.

The worker still uses tokens. Isolation does not make computation free; it keeps investigative noise out of later parent requests. It also adds a model call, coordination latency, and another failure boundary. Use it for context containment or a genuinely separate capability, not merely because a prompt is long.

Just-in-time retrieval: fetch evidence at the moment of use

Retrieval finds relevant information in an external store and adds only that information to the current context. For an agent, the store might contain policies, customer records, previous cases, or long-term memory.

Do not preload an entire manual. Search for the passages relevant to the current decision, then include their source and freshness. Retrieval is not “search once at the beginning”: after a payment timeout, the agent may need retry rules and the idempotency record rather than the eligibility policy.

Bound the result. Filter by tenant, product, region, and effective date, and retrieve a small number of high-quality passages. Ten vaguely related chunks can create contradictory wording. External memory and current context serve different jobs. Agent memory stores information that may matter later; the context window holds what matters for this decision. Retrieving every historical preference into every turn merely moves the bloat elsewhere.

Assemble context as a budget

A production agent should build each request deliberately rather than append messages forever. One workable order is:

1. stable instructions and the output contract
2. current task state, preferably structured
3. a compact summary of earlier work
4. the most recent turns
5. only the retrieved evidence needed for this action

Set a maximum for each layer and reserve output space before filling input space. Every item entering the context should have a reason; “it was in the previous request” is not one. A tool adapter can truncate long logs, store the complete response externally, and provide a retrieval handle. A state reducer can replace ten conversational statements with refund_status: pending.

Measure the assembled prompt in production. Log token counts by layer, compaction events, retrieval results, truncation, and tool-call reasons. Connect those measurements to task outcomes: a falling token count is not success if the agent misses policy exceptions. Observability and tracing makes that relationship visible.

Failure modes you can see first

The agent repeats a question it already asked. Inspect the final assembled context for truncation or a summary that omitted the answer. Keep recent turns, increase the summary’s state fields, and make truncation an explicit metric.

The agent confidently uses the wrong policy version. Retrieval may have returned a stale document, or two versions may be present. Filter by effective date and scope, and put the source ID and date beside each passage. Allow “unknown” when no current policy is found.

The worker returns a polished paragraph the supervisor cannot use. The isolation contract is underspecified. Require status, facts, evidence IDs, side effects, and next action, then validate the result before handing it to the parent.

Answers degrade after each compaction. The summary may be summarizing another summary, becoming vague or preserving an early mistake. Regenerate periodically from a trusted state record and recent raw turns. Exact financial, permission, and side-effect state belongs in structured storage.

The context is small, but the bill and latency remain high. Context engineering does not eliminate retrieval calls, worker calls, summary calls, or large outputs. Measure total tokens across the run, not only the final prompt. Prompt caching can change the economics, but it does not excuse irrelevant context.

When not to use it

A three-turn assistant with a 2,000-token prompt does not need a compaction service. Add the machinery when there is a measurable problem: long sessions, large tool outputs, repeated documents, rising input cost, or quality loss.

Do not use a summary as an audit trail. Keep original messages, tool responses, and side-effect records in durable storage when a human may need to reconstruct what happened. Do not use retrieval to hide a deterministic workflow. If the sequence is known in advance, explicit state is easier to test than an agent searching for its next instruction.

Quick check

Next

Keeping context lean pairs with seeing what the agent actually did — observability and tracing — and bounding spend — cost and latency control.