Self-improving agents: what actually improves, and what just sounds like it

Ask an agent to check its own work and it will find something to change. It nearly always will — not because it spotted a real flaw, but because “review this and improve it” is an instruction, and a model that is asked to produce a critique produces a critique.

That is the whole trouble with the phrase self-improving. The loop looks like learning. Output, evaluate, revise, repeat. But if the evaluator is the same model that wrote the output, with the same context and the same blind spots, then the second pass is just another sample from the same distribution — often a more confident one, because it now has a first draft to agree with.

Useful self-improvement always has the same hidden ingredient: information the model did not have when it wrote the first answer. Once you look for that ingredient, the whole landscape sorts itself out.

The taxonomy, sorted by where the information comes from

Self-critique in context. “Here is your answer, find problems.” Cheapest, most popular, weakest. It helps in one specific case: when the critique step has access to something the drafting step did not — a longer budget to think, a checklist, or, crucially, an error message. On a pure reasoning task with no new input, expect small gains and occasional regressions, because the model will “fix” things that were already right.

Retry after an external verdict. The agent runs its code, the tests fail, the failure text goes back into context, it tries again. This genuinely works, and it is worth being clear about why: a stack trace is information the model could not have derived by thinking harder. The loop is not reflective, it is empirical.

Reflexion-style memory. Same as above, with one addition: after the failure, the agent writes a short lesson in natural language — “the date parser rejects two-digit years; normalise first” — and stores it. On a later, similar task the lesson is retrieved into context. The improvement is real and it persists across episodes, but notice where it comes from: an external verdict produced it, and a retrieval system carried it forward. The model supplied phrasing.

Population and selection. Generate many variants — of a prompt, a plan, a whole program — score them with a grader, keep the winners, mutate. This is the most reliably “self-improving” shape in practice, and it is simply search. The improvement lives entirely in the selection step. With a strong grader it compounds impressively; with a weak one it optimises the grader’s flaws.

Distillation back into weights. Collect the trajectories that succeeded, train on them. This is the only mechanism in the list where the agent is better tomorrow, in a fresh session, with an empty context. Everything else is a better scaffold around an unchanged model.

The ceiling is your verifier, not your model

The pattern across all five: improvement is bounded by the quality of the signal you can produce about whether an attempt was good.

That leads to a question far more useful than “how do I make the agent reflect better”:

What is my source of truth, and how cheaply can I query it?

Rank your tasks by that answer and the engineering falls out.

Source of truth	Example	What improves
A program	tests, a schema, a formal check	Nearly everything. Generate more, keep what passes
The environment	HTTP status, a stack trace, a diff	Retry loops work well; store the lesson
A human	review, thumbs, an accepted edit	Slow and precious — spend it on evaluation, not on labels
The model itself	”rate your answer 1–10”	Almost nothing. It rates its own work highly

The last row is the one that quietly sinks projects. A model scoring its own output correlates with fluency far more than with correctness, so the loop converges on answers that sound better. You will watch your score climb and your users stay unhappy.

What a loop that compounds looks like

Put together, the architecture that actually accumulates capability has four parts, and only the first is glamorous:

1. An attempt. The agent does the task.
2. A verdict from outside the model. Run the thing. Check it. Diff it against expected. This step decides how far the whole system can go, so it deserves most of your engineering.
3. A durable lesson. On failure, write down what went wrong in a form that can be retrieved next time — keyed by task shape, not by exact wording, or it will never match again.
4. Occasional consolidation. Periodically fold the accumulated successes back into the model or into a curated skill set. Without this, the memory grows forever and retrieval gets worse, which is its own kind of decay.

Most agent frameworks give you (1) for free, make (3) easy, and leave (2) and (4) entirely to you — which is exactly backwards from where the value is.

A loop you can actually run

Here is the smallest thing that is genuinely a self-improving agent rather than a self-flattering one. It is a coding task, because the verifier is free — the tests either pass or they do not.

import hashlib, json, sqlite3, subprocess

db = sqlite3.connect("lessons.db")
db.execute("""CREATE TABLE IF NOT EXISTS lessons (
    key TEXT, lesson TEXT, wins INT DEFAULT 0, losses INT DEFAULT 0)""")

def situation_key(task: dict) -> str:
    """Key by the SHAPE of the problem, not its wording — otherwise a lesson
    learned on 'parse invoice dates' never fires on 'read receipt dates'."""
    shape = {"tool": task["tool"], "entity": task["entity"], "op": task["op"]}
    return hashlib.sha1(json.dumps(shape, sort_keys=True).encode()).hexdigest()[:16]

def recall(key, limit=3):
    rows = db.execute(
        "SELECT rowid, lesson FROM lessons WHERE key=? "
        # a lesson that keeps preceding failures has earned eviction, not retrieval
        "ORDER BY (wins - losses) DESC LIMIT ?", (key, limit)).fetchall()
    return rows

def verify(code: str) -> tuple[bool, str]:
    """The external signal. Nothing here asks the model what it thinks."""
    r = subprocess.run(["python", "-m", "pytest", "-q", "--tb=short"],
                       input=code, capture_output=True, text=True, timeout=60)
    return r.returncode == 0, (r.stdout + r.stderr)[-2000:]

def attempt(task, max_tries=3):
    key = situation_key(task)
    used = recall(key)
    notes = "\n".join(f"- {l}" for _, l in used)

    for i in range(max_tries):
        code = llm(f"{task['prompt']}\n\nKnown pitfalls:\n{notes or '(none yet)'}")
        ok, output = verify(code)

        if ok:
            # Credit the lessons that were in context when it worked.
            for rowid, _ in used:
                db.execute("UPDATE lessons SET wins=wins+1 WHERE rowid=?", (rowid,))
            db.commit()
            return code

        # Failure. The lesson is written FROM THE VERDICT, not from introspection.
        for rowid, _ in used:
            db.execute("UPDATE lessons SET losses=losses+1 WHERE rowid=?", (rowid,))
        lesson = llm(
            "A test failed. In one sentence, state the pitfall a future attempt "
            f"should avoid. Do not restate the task.\n\nFailure:\n{output}")
        db.execute("INSERT INTO lessons (key, lesson) VALUES (?,?)", (key, lesson))
        db.commit()
        notes += f"\n- {lesson}"

    return None

Four things in there are doing the real work, and none of them are the prompt:

1. verify() runs the code. The improvement signal is a process exit code. Swap it for a schema check, an HTTP status, or a diff against expected output — but it must come from outside the model.
2. The lesson is written after the verdict, with the failure text in context. The model is phrasing something it has just been shown, not inventing a critique of work it already believes in.
3. situation_key keys by shape. Tool, entity, operation — not the prose. This is the difference between memory that fires when it is needed and memory that fires when the wording happens to rhyme.
4. wins and losses per lesson. Every retrieval is scored by what happened next. A lesson learned from a misdiagnosis accumulates losses and sinks below the retrieval cut on its own.

That last column is the whole defence against memory poisoning, and it costs one integer.

What to watch once it is running

Log these three from the start, because each one catches a different way the loop can look healthy while getting worse:

# 1. Is the memory earning its context?     → compare against notes="" runs
# 2. Are we improving or narrowing?          → pass@1 vs pass@16 on a fixed set
# 3. Which lessons are toxic?                → SELECT * WHERE losses > wins

If pass@1 climbs while pass@16 is flat, stop and look: the agent is not getting better, its distribution is getting tighter. That is the failure mode below, and it is the one that hides inside a rising dashboard.

Three failure modes worth naming

Self-congratulation. The agent’s own confidence rises with each revision regardless of correctness. If you use model-scored quality as your metric, you will measure this and mistake it for progress. Always keep one metric the agent cannot influence.

Memory poisoning. A lesson learned from a misdiagnosed failure gets stored and retrieved forever, and now every similar task starts with a false premise. Memory needs the same hygiene as a cache: an entry should be able to expire, and a lesson that keeps preceding failures should be evicted. Track which memories are retrieved before successes versus before failures.

Improvement that is really narrowing. The loop converges on one template that reliably scores well and stops exploring. Success rate on your benchmark rises; anything slightly unusual breaks. The tell is pass@1 climbing while pass@k stays flat — the agent is not getting better, its distribution is getting tighter.

The honest summary

An agent cannot lift itself by reviewing its own work, for the same reason you cannot proofread your way to knowing a fact you never learned. What it can do — and what the good systems all do — is arrange to be told it was wrong, cheaply and often, and then keep what that told it.

So when someone describes a self-improving agent, the only question worth asking is: what tells it that it failed? If the answer is “the model decides”, it is a more expensive way to get the same answer. If the answer is a test suite, a compiler, a diff, or a user who came back, then there is something real there — and the interesting engineering is in making that signal cheap enough to collect a thousand times a day.