Agent Memory — Working, Episodic & Semantic

The honest version of this topic is shorter than the marketing version. Memory in an agent is just state that survives — across turns, or across sessions. The interesting question is which state survives, how long, and where it lives. The field borrowed a taxonomy from cognitive science to name the answers, and that taxonomy — working, episodic, semantic, procedural — is now the reference vocabulary, anchored by the CoALA paper (Cognitive Architectures for Language Agents, 2023).

Four kinds of memory

Read the diagram by lifespan, not by name. Working memory is the only ephemeral one — it is the live context the model reasons over this turn, and it is gone when the session ends. The other three persist. Episodic logs what happened (events, trajectories) and is recalled by similarity later. Semantic holds durable facts — the user’s name, their preferences — the de-contextualized things you want true tomorrow. Procedural is the agent’s how-to: mostly its code and prompts, plus whatever is implicit in the model weights. CoALA stresses procedural memory “must be initialized by the designer” — it is foundational and risky to rewrite at runtime, so most systems treat it as static.

A common pattern links the durable two: repeated episodic observations get consolidated into one semantic fact. The agent notices “user corrected the date format three times” (three episodes) and abstracts it into a single durable preference: “user prefers DD/MM/YYYY.”

The production reality: working memory is checkpointed state

Here is the part the cognitive-science framing obscures. In real systems, “working memory” is not a special biological buffer. It is checkpointed agent state — and you have already met it. In LangGraph persistence, short-term memory is thread-scoped state: a checkpointer (InMemorySaver, PostgresSaver, Redis) snapshots the graph after each node and restores it on the next invoke for the same thread_id. Start a new thread_id and that state vanishes. That is working memory: ephemeral in spirit, but durably snapshotted so the agent can resume mid-conversation.

This is what the overwhelming majority of agents actually rely on day to day. The elaborate stack often is not needed.

Memory is not RAG

This is the distinction that trips everyone up, so be precise about it.

RAG retrieves external documents at query time to answer a question, then forgets. It is stateless and read-only at query time: pull the top-k relevant chunks from an index, stuff them in context, answer “what does the document say?”, done. (The retrieval machinery — embeddings and vector search — is shared with memory, which is why the two get conflated.)

Memory persists the agent’s own state, history, and learned facts across sessions. Crucially, memory has a distinct write phase that RAG lacks: extract a fact, decide whether to store it, store it — and only then, later, read it back. Letta puts it bluntly: retrieval “is a tool for agent memory, [but] it is not memory in of itself.”

They are not rivals. A production agent often uses RAG to answer questions about a knowledge base and memory to personalize across sessions.

See it: run a session, then start a new one

The widget below is the whole lesson in miniature. Run a short sequence of interactions, watch which panel each piece of information lands in and how the context-window meter fills with working memory — then hit Start new session and see the split: working memory clears, the durable stores survive, and a retrieval step pulls them back into the fresh context window.

The aha: not all memory is the same. “Maya’s name” went to a durable semantic slot because you want it true forever; “booked a flight to Tokyo” went to episodic because it is an event you might recall later; the raw turns sat in working memory and evaporated the moment the session ended. Pick the memory type for the durability you need — and notice how little of it the durable side actually has to hold.

The vendor landscape (without the hype)

Three names come up constantly. They are genuinely good engineering; the trap is reaching for them before you need them.

A few honest caveats. The “context window = RAM, external store = disk” analogy is MemGPT’s, and it is genuinely useful. Zep reports edging out MemGPT on the Deep Memory Retrieval benchmark (94.8% vs 93.4%) and large gains on the harder LongMemEval — but those are the vendor’s own paper figures, not independently reproduced here, so read them as “Zep reports,” not as neutral fact. For the open-source primitives, LangGraph/LangMem give you the two pieces directly: thread-scoped checkpointers (working memory) and a cross-thread store with namespaces and optional semantic search (long-term memory). Anthropic and OpenAI both ship first-party memory features too — Anthropic’s file-based memory tool lets the model keep a NOTES.md-style file in a /memories directory that persists across sessions (public beta).

If you want the longer tour of these architectures, the blog post the three kinds of memory production agents actually use goes deeper on the trade-offs.

Governance: the actually-hard part

Storing a fact is easy. The hard problems are what to remember, when to forget, and how to correct.

• What to remember. You cannot keep everything — promote only salient facts. Mem0’s LLM-judged ADD/UPDATE/DELETE/NOOP is exactly this selective-promotion decision made automatically.
• Correction & expiry. Better systems version rather than destroy. Zep invalidates stale edges via validity intervals (keeping the history); Mem0 issues UPDATE/DELETE. Blind overwrite loses the audit trail.
• Context bloat. Stuffing the window with retrieved memories leaves the model less room to reason and measurably degrades quality. This is why the frontier mantra is context engineering — keep the minimal set of high-signal tokens, retrieve just in time. Bigger context windows shift this trade-off; they do not solve the governance problem.

Quick check

Next

You now have the taxonomy and, more importantly, the judgment to use less of it than the hype implies. Two threads continue from here: the engineering of durable state lives in LangGraph persistence (checkpointers and stores), and the risk lives in memory poisoning — because the moment memory persists, it becomes something an attacker wants to write to.