Agent Memory — Working, Episodic & Semantic

The honest version of this topic is shorter than the marketing version. Memory in an agent is just state that survives — across turns, or across sessions. The interesting question is which state survives, how long, and where it lives. The field borrowed a taxonomy from cognitive science to name the answers, and that taxonomy — working, episodic, semantic, procedural — is now the reference vocabulary, anchored by the CoALA paper (Cognitive Architectures for Language Agents, 2023).

Four kinds of memory

Read the diagram by lifespan, not by name. Working memory is the only ephemeral one — it is the live context the model reasons over this turn, and it is gone when the session ends. The other three persist. Episodic logs what happened (events, trajectories) and is recalled by similarity later. Semantic holds durable facts — the user’s name, their preferences — the de-contextualized things you want true tomorrow. Procedural is the agent’s how-to: mostly its code and prompts, plus whatever is implicit in the model weights. CoALA stresses procedural memory “must be initialized by the designer” — it is foundational and risky to rewrite at runtime, so most systems treat it as static.

A common pattern links the durable two: repeated episodic observations get consolidated into one semantic fact. The agent notices “user corrected the date format three times” (three episodes) and abstracts it into a single durable preference: “user prefers DD/MM/YYYY.”

The production reality: working memory is checkpointed state

Here is the part the cognitive-science framing obscures. In real systems, “working memory” is not a special biological buffer. It is checkpointed agent state — and you have already met it. In LangGraph persistence, short-term memory is thread-scoped state: a checkpointer (InMemorySaver, PostgresSaver, Redis) snapshots the graph after each node and restores it on the next invoke for the same thread_id. Start a new thread_id and that state vanishes. That is working memory: ephemeral in spirit, but durably snapshotted so the agent can resume mid-conversation.

This is what the overwhelming majority of agents actually rely on day to day. The elaborate stack often is not needed.

Memory is not RAG

This is the distinction that trips everyone up, so be precise about it.

RAG retrieves external documents at query time to answer a question, then forgets. It is stateless and read-only at query time: pull the top-k relevant chunks from an index, stuff them in context, answer “what does the document say?”, done. (The retrieval machinery — embeddings and vector search — is shared with memory, which is why the two get conflated.)

Memory persists the agent’s own state, history, and learned facts across sessions. Crucially, memory has a distinct write phase that RAG lacks: extract a fact, decide whether to store it, store it — and only then, later, read it back. Letta puts it bluntly: retrieval “is a tool for agent memory, [but] it is not memory in of itself.”

They are not rivals. A production agent often uses RAG to answer questions about a knowledge base and memory to personalize across sessions.

Watch the session boundary

Trace one concrete story across two sessions. In session 1 the user says “my name is Maya,” books a flight to Tokyo, and chats — all of it sitting in working memory, the live context window. As the salient facts surface, the agent writes the durable ones out: “Maya prefers aisle seats” to semantic, “booked a Tokyo flight” to episodic. Then the session ends.

When session 2 opens on a new thread_id, working memory is empty — the live context cleared with the old session. The durable stores survive, and a retrieval step pulls the relevant facts back into the fresh window, so the agent greets Maya by name without her repeating it.

The aha: not all memory is the same. “Maya’s name” went to a durable semantic slot because you want it true forever; “booked a flight to Tokyo” went to episodic because it is an event you might recall later; the raw turns sat in working memory and evaporated the moment the session ended. Pick the memory type for the durability you need — and notice how little of it the durable side actually has to hold.

The vendor landscape (without the hype)

Three names come up constantly. They are genuinely good engineering; the trap is reaching for them before you need them.

A few honest caveats. The “context window = RAM, external store = disk” analogy is MemGPT’s, and it is genuinely useful. Zep reports edging out MemGPT on the Deep Memory Retrieval benchmark (94.8% vs 93.4%) and large gains on the harder LongMemEval — but those are the vendor’s own paper figures, not independently reproduced here, so read them as “Zep reports,” not as neutral fact. For the open-source primitives, LangGraph/LangMem give you the two pieces directly: thread-scoped checkpointers (working memory) and a cross-thread store with namespaces and optional semantic search (long-term memory). Anthropic and OpenAI both ship first-party memory features too — Anthropic’s file-based memory tool lets the model keep a NOTES.md-style file in a /memories directory that persists across sessions (public beta).

If you want the longer tour of these architectures, the blog post the three kinds of memory production agents actually use goes deeper on the trade-offs.

Governance: the actually-hard part

Storing a fact is easy. The hard problems are what to remember, when to forget, and how to correct.

• What to remember. You cannot keep everything — promote only salient facts. Mem0’s LLM-judged ADD/UPDATE/DELETE/NOOP is exactly this selective-promotion decision made automatically.
• Correction & expiry. Better systems version rather than destroy. Zep invalidates stale edges via validity intervals (keeping the history); Mem0 issues UPDATE/DELETE. Blind overwrite loses the audit trail.
• Context bloat. Stuffing the window with retrieved memories leaves the model less room to reason and measurably degrades quality. This is why the frontier mantra is context engineering — keep the minimal set of high-signal tokens, retrieve just in time. Bigger context windows shift this trade-off; they do not solve the governance problem.

In one breath

• “Memory” is just state that survives — across turns or sessions; the real questions are which state survives, how long, and where it lives.
• Four types, read by lifespan: working (ephemeral live context = checkpointed thread state), episodic (durable event log, recalled by similarity), semantic (durable facts/preferences), procedural (the agent’s code/prompts/weights — foundational, designer-set).
• Memory is not RAG: RAG reads external documents at query time and forgets; memory persists the agent’s own state with a distinct write phase (extract → decide → store), then reads it back later.
• The honest thesis: most agents need only working memory plus a thin slice of semantic — full episodic + semantic + graph + procedural stacks (Letta, Mem0, Zep) are frequently over-engineering.
• The hard part is governance — what to remember, when to forget/correct (version, don’t blind-overwrite), and context bloat — and durable memory is an attack surface (poisoning persists across sessions).

Quick check

Next

You now have the taxonomy and, more importantly, the judgment to use less of it than the hype implies. Two threads continue from here: the engineering of durable state lives in LangGraph persistence (checkpointers and stores), and the risk lives in memory poisoning — because the moment memory persists, it becomes something an attacker wants to write to.