What types of memory do agents use, and what is context engineering and compaction?

How to think about it

Agents typically use transient working memory—the messages, instructions, tool results, and retrieved documents in the current context window, the finite token budget visible to one model call—and durable external memory kept by the application. Context engineering decides what enters that window and in what form; compaction compresses older context into a smaller state when the working set grows too large.

Picture a support agent at 3:07 a.m. A customer, C-1842, says, “The blender I bought last month arrived cracked. Can I get a refund?” The agent must find the right order, check the damaged-item policy, avoid exposing another customer’s data, and remember what has already been promised. Sending the entire conversation and every company document is not a solution. That is how a useful fact gets buried under 20 pages of tool output.

Why the distinction matters

An LLM does not automatically carry memory from one normal API call to the next. In a stateless setup, the application must resend the relevant instructions, history, and tool results each time. If it does not, the agent cannot use them, even if a previous turn felt remembered to the user.

Memory is therefore an application pipeline, not a mystical property of the model. The system writes useful facts or events to storage, retrieves candidates on a later request, filters them for tenant and permission boundaries, and injects a small selected set into the current context. Every step can fail: a bad write creates stale memory, bad retrieval supplies the wrong customer, and an oversized injection leaves less room for the current task.

Common misconception: the context window is not long-term memory. It is more like a desk. The model can work with what is placed on the desk for this call, but the desk is cleared unless the application saves and restores information elsewhere.

The main memory types

A practical taxonomy has one short-term category and three common long-term categories:

Type	What it holds	Support-agent example
Working or short-term memory	The current prompt, recent turns, tools, and results	The customer’s latest request and the order lookup result
Episodic memory	Time-stamped events or experiences	“Customer uploaded a damage photo on August 28”
Semantic memory	Durable facts, preferences, and relationships	“C-1842 prefers email” or “Order O-7719 belongs to C-1842”
Procedural memory	Instructions for how to perform a task	“Check ownership, check the return window, then call the refund tool”

These labels describe the kind of information, not necessarily four separate databases. One record can be both episodic and semantic. A memory system might store facts in a relational database, documents in object storage, and use a vector database for retrieval. A vector database indexes embeddings, which are numerical representations of meaning, so it can find text similar to a query. It does not decide whether a memory is true, current, relevant, or authorized.

Procedural memory deserves special care. In production, critical procedures should usually live in versioned code, tool definitions, or authoritative documents rather than only in a model-generated summary. A summary that changes “ask for approval” into “approve automatically” is not a small mistake.

What context engineering means

Context engineering is the deliberate design of the model’s working set. It includes retrieval, filtering, summarization, ordering, formatting, tool-output shaping, and state management. It is broader than writing a clever system prompt.

The goal is not to maximize the amount of text. The goal is to maximize the amount of useful, trustworthy information per token. Selection should consider:

• relevance to the current task;
• authority, so the current refund policy beats an old conversational claim;
• recency, so the latest order status beats last month’s status;
• permissions, so a customer’s records cannot cross into another tenant;
• dependencies, so an order ID travels with the facts that explain it.

Ordering matters too. The system instructions and safety constraints normally come first, followed by the current task state, authoritative evidence, recent conversation, and only the tool definitions needed for this step. A long tool schema or irrelevant policy manual consumes context and can bury the fact the agent needs. More context can produce less reliable answers.

A concrete example

Suppose the agent has a total context budget of 32,000 tokens. A token is a small chunk of text used by the model for context and generation. The application reserves 4,000 tokens for the answer and tool calls, leaving a safe input budget of 28,000.

The naive prompt contains:

• system instructions and authorization state: 1,500 tokens;
• every available tool schema: 4,000 tokens;
• the entire returns manual: 5,000 tokens;
• the complete conversation and tool transcript: 19,000 tokens.

That is 29,500 input tokens before the reserved output. The request may be rejected, truncated, or leave the model with no room to complete the task.

A context-engineered request could instead contain the system and authorization state, the two needed tools, the one relevant policy section, the last six turns, and a structured state record. Suppose those consume 1,500, 1,100, 700, 6,000, and 400 tokens respectively: 9,700 input tokens in total. The agent now has room to reason and respond. It also avoids retrieving C-1842’s unrelated shipping preferences because they do not help decide this refund.

What compaction does

Compaction is the controlled replacement of older conversation and tool history with a smaller representation of the information still needed. It may use a model-generated summary, deterministic extraction, or both. It is usually triggered by token budget or trajectory length, not by a fixed number of messages: one tool response can be larger than ten ordinary turns.

A good compactor preserves the goal, entities, dates, amounts, decisions, constraints, unresolved work, and provenance. It removes greetings, repeated instructions, and raw tool output whose important fields have already been extracted.

For the blender case, a compact state might look like this:

{
  "goal": "Refund damaged blender",
  "customer_id": "C-1842",
  "order_id": "O-7719",
  "facts": [
    "Order delivered 2026-08-02 for $89",
    "Damage photo uploaded 2026-08-28"
  ],
  "status": "Refund not yet issued",
  "constraints": [
    "30-day damaged-item policy applies",
    "Verify the payment destination"
  ],
  "sources": [
    "order-service:O-7719",
    "returns-policy:damaged-items"
  ],
  "open_task": "Confirm refund rather than replacement"
}

The raw transcript should remain in an audit store. The compact state is only the model’s working representation. Compaction itself can cost a model call and can introduce a false detail, so critical fields such as order status, payment amount, and authorization should be re-read from their source systems rather than trusted solely because a summary says so.

The trade-off and a failure mode

Compaction is lossy. It can drop the exact wording of a customer’s consent, a policy exception, or a tool error. For legal, financial, or safety-critical workflows, keep the authoritative record outside the context and preserve source identifiers in the compact state. Use the summary for narrative continuity, not as the database of record.

Long-term memory also creates privacy and staleness risks. A customer’s old address may no longer be valid. A retrieved memory may belong to a different workspace. Memory writes should be selective, access-controlled, time-stamped, and sometimes subject to expiration or user deletion. For a one-turn question with a 2,000-token prompt, adding a full memory subsystem may only add latency and new failure modes. Do not use it merely because the architecture diagram has an empty box labelled “memory.”

A common production symptom is an agent that works for ten turns and then says a refund was already approved, or calls the refund tool twice. Inspect the actual assembled context and retrieved records first. The cause is often a compaction summary that lost “not yet issued,” a stale memory outranking the order service, or retrieval without a tenant filter. Logging memory reads, writes, source IDs, timestamps, and the final prompt makes this diagnosable.

What they’ll ask next

Is a vector database the agent’s long-term memory?
No. It is one way to retrieve semantically similar records. The application still needs schemas, metadata filters, authorization, freshness rules, conflict handling, and a source of truth.

What must a compaction summary preserve?
The current goal, named entities, exact IDs, important dates and numbers, decisions, constraints, pending actions, and links to authoritative sources. It should not turn “the customer requested a refund” into “the refund was approved.”

How would you evaluate an agent memory system?
Test recall of information that should be remembered, precision so irrelevant memories are not injected, resistance to stale or conflicting facts, permission isolation, latency, and token cost. Replay conversations where the correct answer depends on an event several turns earlier, then test that the agent still refuses to use another tenant’s record.

Say this in the interview: Agents have bounded context as working memory and external stores for durable episodic, semantic, and procedural information; context engineering selects the safest useful working set, while compaction compresses old state without losing facts, decisions, constraints, or provenance.