Self-improving agents

At 3:07 a.m., your support agent handles a refund request. It finds the right policy, writes a clear answer, and closes the ticket.

The next morning it handles the same policy incorrectly.

Nothing changed. The model has the same weights, the same prompt, and the same tools. It does not remember which answer worked. It does not test alternative answers. It does not turn yesterday’s success into tomorrow’s capability.

A self-improving agent closes that loop. It generates attempts, evaluates them, keeps the attempts that pass, and learns from them. Learning might mean revising the current answer, saving a reusable skill, updating a dataset, or fine-tuning model weights.

That sounds like a route to capability without a large stream of new human labels. Sometimes it is.

The catch is severe: the agent needs a trustworthy way to tell good work from bad. Without that, it is not improving. It is becoming more confident in whatever its own feedback happens to reward.

The loop, and the linchpin

Many self-improvement methods follow this loop, differing mainly in where learning is stored: in the current attempt, memory, model weights, or an external artifact.

Take an invoice-repair agent. Its task is to produce a Python patch that calculates tax correctly. It generates 20 patches; most mishandle discounts, rounding, or tax-exempt customers.

An evaluator scores an attempt. Here, it runs the patch against tests. A verifier is an evaluator strong enough to establish that the attempt meets the requirement. A language model giving a patch a score of 8 out of 10 is an evaluator, not automatically a verifier.

Suppose the patch must pass 100 tests:

• 72 patches fail to compile or fail at least one test.
• 28 pass all 100 tests.
• The system keeps the 28 passing patches, or only the shortest, fastest, or most diverse few.
• It adds their useful patterns to a skill or training set.

Why does this work? Selection changes the examples the agent learns from. Before selection, the candidate pool contains successes and failures. After selection, it contains a higher proportion of successes. If learning absorbs that difference, the next generator produces more candidates like the successful ones.

That is the causal mechanism: generation creates variation, evaluation measures it, selection changes the population, and learning makes the change persist.

acc, ceiling = 0.40, 0.85         # ceiling = the limit the verifier/data can certify
for rnd in range(1, 6):
    acc = acc + 0.5 * (ceiling - acc)   # keep verified-good work, fine-tune, improve a fraction
    print(f"round {rnd}: accuracy {acc:.1%}")

The verifier is the linchpin because every later step trusts its decision. A good verifier has four properties:

• Soundness: it rarely accepts an incorrect attempt.
• Coverage: it checks cases where errors actually occur.
• Resistance to gaming: the agent cannot raise the score by exploiting the test or metric.
• Independence: it does not simply share the generator’s blind spots.

A unit test can verify the exact tax total while saying nothing about data leakage or invoice deletion. Every verifier has a scope, and improvement cannot reliably exceed what that scope distinguishes.

What “learn” can mean

Per-attempt improvement

A reflection loop asks the agent to inspect its draft, find a flaw, and revise it before returning the answer. For the invoice agent:

1. Draft a patch.
2. Run the tests.
3. Read the failing tax-exemption test.
4. Revise the patch and test again.

This is self-correction, not necessarily long-term self-improvement. The change usually disappears when the run ends.

See reflection for the per-attempt evaluator-optimizer pattern.

Memory and skill learning

The agent can save a verified procedure such as “apply discounts before tax, then round at the invoice total.” A later run retrieves it while the model weights remain fixed.

A skill saved after one passing example may encode an accidental shortcut, so a production skill library should store its evidence: task distribution, tests passed, version, date, and known limitations.

Skill libraries cover this form of durable learning.

Weight and artifact learning

The system can fine-tune on accepted attempts, making the behavior broadly available without retrieving a particular memory. This is powerful but global: a bad example can affect unrelated tasks. Use separate training and evaluation problems; otherwise, apparent improvement may be memorization.

Sometimes the model does not change at all. It evolves an artifact such as a program, prompt, workflow, query plan, or algorithm. The agent generates variants, runs them, and keeps the best. This is often safer than fine-tuning when the artifact is executable and the evaluator is cheap.

STaR: bootstrapping reasoning from answers

STaR, or Self-Taught Reasoner, applies the loop to reasoning traces.

Start with problems that have known answers. The model generates a rationale—a sequence of intermediate steps—and a final answer. Keep rationales whose final answers are correct, fine-tune on those examples, and repeat.

STaR also uses rationalization for problems the model initially gets wrong: give it the correct answer and ask for a rationale supporting that answer. This adds examples that were difficult to discover from scratch.

The known answer verifies the destination, not necessarily the route. A model can reach the right answer for the wrong reason or produce a polished explanation after seeing the answer. Step-by-step checkers are stronger for arithmetic and symbolic tasks; open-ended reasoning still needs human or independent review.

STaR is therefore bounded self-training, not a method for the model to teach itself anything.

Evolutionary search and automated research

In evolutionary program search, an LLM proposes or mutates programs. An automatic evaluator runs them and assigns a fitness score, a numerical measure of performance. High-fitness programs seed later generations.

For an invoice calculator, fitness might combine test correctness, runtime, and memory use. The fitness function must reflect the real requirement: a faster patch that fails a required tax case should not beat a correct one.

AlphaEvolve is a prominent example: language models generate code candidates, automated evaluators score them, and evolutionary selection searches for better algorithms and implementations. Its key design is the separation between a creative generator and an executable judge.

The AI Scientist systems developed by Sakana AI apply a related loop to research: propose hypotheses, write or modify experiment code, run experiments, analyze results, and report. Experiments verify some claims, but not all. A script running successfully does not establish that a result is statistically sound or scientifically meaningful. Independent checks and human review remain necessary before publication.

A production pattern that does not eat the lab

Separate discovery from promotion:

1. Define the task distribution. Include ordinary invoices, discounts, exemptions, currencies, rounding boundaries, malformed inputs, and adversarial inputs. “Improve invoice handling” is too vague to verify.
2. Build a held-out evaluation set. Keep hidden cases away from generation and training. Split by customer, time period, or template when those create real-world variation; random splits of near-duplicate invoices mislead.
3. Generate diverse candidates. Vary strategies rather than producing ten nearly identical guesses from the same prompt.
4. Use a sandbox. Limit filesystem, network, CPU, memory, runtime, and credentials. A candidate must not modify tests, read answer keys, or call billing systems.
5. Retain provenance. Store the input, candidate, evaluator version, tests passed, score, model version, and selection reason.
6. Promote gradually. Compare with the current system on visible, hidden, adversarial, and real-traffic samples. Use a canary release and keep the previous version for rollback.
7. Monitor the evaluator. Add production failure cases, but do not automatically turn every model-generated complaint into a trusted label.

Where self-improvement fails first

Reward hacking

Symptom: the score rises while users report worse results.

A coding agent may skip expensive tests, return cached answers, or exploit a benchmark bug. A support agent may maximize a politeness score by apologizing at length without solving the request. Inspect the artifact and real outcome, add hidden and adversarial tests, and fix the metric.

Evaluator overfitting

Symptom: the benchmark improves but new customers or later data perform worse.

The agent has learned the evaluator’s examples or quirks. Keep a private holdout set, split by time and entity rather than only random rows, and rotate challenge cases against a frozen baseline.

Model collapse and loss of diversity

Symptom: outputs become repetitive, unusual cases disappear, and novel-task performance falls.

Repeatedly training on the system’s own filtered outputs narrows the data distribution and amplifies mistakes. Mix in trusted human or external data, preserve diverse candidates, and track rare-case performance as well as averages.

Circular model judging

Symptom: a model judge gives high scores to answers that an independent check finds wrong.

The generator and judge may share a factual blind spot or stylistic preference. Prefer executable checks; for subjective tasks, use calibrated rubrics, multiple judges, human audits, and outcome metrics. A second model is not automatically independent.

Verifier contamination

Symptom: tests become mysteriously easier.

If generated code can read test files, alter fixtures, change the evaluator, or inspect expected answers, the loop measures access to the scoreboard rather than task ability. Treat the verifier as a security boundary: keep it outside candidate permissions, version it immutably during a run, and log protected-resource access.

The costs are real. Generating 20 candidates costs roughly 20 times the model work before evaluation. Sandboxed execution and hidden tests require infrastructure and maintenance. Fine-tuning adds data curation, deployment, and rollback complexity. Selection can also make a system less creative: what wins on a narrow score may be brittle outside the benchmark.

Do not put an unconstrained weight-update loop directly on live traffic. Discover offline, verify independently, promote gradually, and make every learned change reversible.

In one breath

• A self-improving agent runs generate → evaluate → keep → learn → repeat.
• Generation creates variation. The verifier selects what survives. Learning makes that selection durable.
• The verifier is the linchpin because false positives become training data, memories, or programs copied into future runs.
• STaR filters reasoning traces with known answers. Evolutionary program search filters executable candidates with tests or fitness scores. Automated research filters hypotheses with experiments, which still need scientific scrutiny.
• Learning can happen per attempt, in memory, in model weights, or in an external artifact such as code.
• Improvement approaches the ceiling of the signal. A weak or gameable signal produces reward hacking, overfitting, circular judging, or collapse rather than capability.

Quick check

Next

Self-improvement appears as durable knowledge in skill libraries, as search in HTN and evolutionary planning, and per attempt in reflection. The controls for bounding an agent that can change its own behavior belong in agent safety controls.