The agent that remembers: episodic memory, and the consolidation nobody builds
Most agent memory is a growing pile of transcripts with a similarity search bolted on. That works for a week and then quietly gets worse, because retrieval quality falls as the pile grows. The part that keeps it working is the part almost nobody implements: periodically turning experience into something smaller.
The first version of agent memory is always the same. Append every episode to a store, embed it, and on the next task retrieve the top-k by cosine similarity. It works immediately, which is the problem — it works well enough that nobody revisits it until it has quietly stopped working.
The failure is not dramatic. There is no error. The agent just gets less useful as it accumulates more experience, which is the exact opposite of what memory was supposed to do.
Why more memory makes retrieval worse
Retrieval quality is not about how much you stored. It is about how distinguishable the right entry is from everything else competing for the same k slots.
Early on, an agent has fifty episodes and they are all quite different. Ask for something about date parsing and one entry is obviously the closest. After ten thousand episodes there are four hundred entries that touch date parsing, mostly routine successes that taught nothing, and the one entry describing the actual pitfall is now competing with 399 near-duplicates whose embeddings are just as close.
You did not lose the lesson. You buried it.
This is the core reason append-only memory decays, and it is why
turning up k does not rescue it: you get more near-duplicates, spend
more context, and crowd out the reasoning space you needed for the task
itself.
Two tiers, because they answer different questions
Borrowing the distinction from human memory is useful here, not because the analogy is deep but because it separates two things that get conflated:
Episodic memory — what happened. A specific run: the task, the approach, the outcome, the verdict. High fidelity, high volume, mostly useful in the days after it happened.
Semantic memory — what it means. A durable rule distilled from many episodes: “invoices from this vendor put the date in DD/MM even though the locale header says US”. Low volume, long-lived, worth injecting into context far more often.
Almost every production memory system is only the first tier, and the second is the one that carries the compounding.
Decide what to keep at write time
The cheapest improvement available to most systems: stop storing everything.
An episode is worth keeping when it carries information the agent could not reconstruct. In practice that means:
- Failures with a verdict. What was attempted, what the external signal said, what the corrected approach was. These are the highest value entries you will ever have.
- Surprises. A success that took an unusual route, or where the obvious approach did not work.
- Stable facts about the environment. Rate limits, schema quirks, which of three endpoints actually returns the field you need.
And it means not keeping the long tail of routine successes. A thousand episodes of “called the API, it worked” contribute nothing on retrieval and actively crowd the index.
If you feel nervous throwing episodes away, keep them in cold storage outside the retrieval index. The point is not to delete history; it is to stop history competing with knowledge.
Key by situation, not by prose
The second common defect is keying on the raw text. Two tasks that are the same problem described differently will not retrieve each other, and two unrelated tasks that share vocabulary will.
A more durable key is a small structured description of the situation: the tool being used, the entity type, the error class, the stage of the workflow. Embed that, or index on it directly, and retrieve by it. Keep the prose as the payload, not as the key.
This is the difference between memory that fires when it is needed and memory that fires when the wording happens to rhyme.
The consolidation pass
Consolidation is a scheduled job, not part of the request path. It reads recent episodes and does four things:
- Cluster near-duplicate episodes.
- Promote a cluster into a single semantic rule when it has enough support — several independent episodes pointing the same way.
- Merge the new rule with any existing rule it subsumes or contradicts, keeping one statement rather than two that disagree.
- Evict episodic entries whose content is now represented by a rule, and rules that have stopped earning retrieval.
Run it nightly, or after every N episodes. It is unglamorous work that looks like database maintenance, which is roughly why it does not get built — and it is the single thing that keeps memory from degrading as the agent gains experience.
The contradiction-handling step in (3) matters more than it looks. Two rules that disagree are worse than no rule, because retrieval will sometimes surface one and sometimes the other, and the agent’s behaviour becomes non-deterministic in a way that is very hard to debug from the outside.
Measure memory the only way that counts
Memory systems are usually evaluated on retrieval metrics — recall@k against a hand-labelled set — which tells you the index works, not that the memory helps.
The measurement that matters is causal and simple to collect: for every retrieved entry, record whether the episode it was retrieved into succeeded or failed. Then look at each entry’s record.
- Entries that appear disproportionately before successes are earning their place.
- Entries that appear disproportionately before failures are actively harmful — a poisoned lesson learned from a misdiagnosis, steering every similar task wrong.
- Entries that are retrieved constantly with no effect either way are paying context rent for nothing.
That single table turns memory from a black box into something you can prune deliberately. Most teams never build it, and so the only signal they have about memory quality is a vague sense that the agent used to be better.
What to build first
If you are starting today and want the shortest path to memory that compounds rather than decays:
Store failures with their verdicts and nothing else. Key them by a structured description of the situation. Run a weekly job that merges duplicates into rules. Log which entries were retrieved before which outcomes, and delete the ones that precede failures.
That is a smaller system than most teams build, and it will still be working in six months, which is more than can be said for the transcript pile.