Recursive self-improvement

Self-improving agents get better at a task through a generate-evaluate-keep loop. Recursive self-improvement (RSI) is the more consequential version: a system that improves its own ability to improve. Each gain makes the next gain easier, so progress compounds instead of adding up — the scenario behind the term intelligence explosion.

Why gains compound

The difference between ordinary and recursive improvement is the difference between adding and multiplying. If improvement scales with current capability, capability grows exponentially:

cap_linear, cap_recursive = 1.0, 1.0
step = 0.3

print("round   linear   recursive")
for r in range(1, 7):
    cap_linear += step                          # fixed improvement each round
    cap_recursive += step * cap_recursive       # improvement scales with current capability
    print(f"  {r}     {cap_linear:5.2f}     {cap_recursive:7.2f}")

round   linear   recursive
  1     1.30        1.30
  2     1.60        1.69
  3     1.90        2.20
  4     2.20        2.86
  5     2.50        3.71
  6     2.80        4.83

Six rounds in, linear progress reached 2.8 while the recursive process hit 4.83 and is pulling away. A better optimizer builds a better optimizer faster — and if nothing slows the loop, the curve goes near-vertical. Whether it actually does, and how fast, is the crux.

Takeoff speed, and the core risk

The decisive question is takeoff speed. Fast takeoff (“FOOM”) is a rapid, near-discontinuous jump once the loop ignites; slow takeoff is gradual and continuous. It matters enormously for safety: a fast takeoff leaves almost no time to notice problems and correct course, so alignment would have to be solved in advance; a slow takeoff offers chances to adjust along the way.

And here is why RSI is the heart of AGI-risk arguments: a recursive self-improvement loop is exactly what you must not have around a misaligned objective. A small reward-hacking or mesa-optimization flaw, amplified through a fast compounding loop with no human in the path, is the canonical catastrophe — an optimizer that gets rapidly more powerful while pursuing the wrong goal.

In one breath

• Recursive self-improvement is a system improving its own ability to improve, so gains compound (multiply, not add) — the intelligence explosion (the demo: recursive 4.83 vs linear 2.8 in 6 rounds).
• The crux is takeoff speed: fast (“FOOM,” near-discontinuous) leaves little time to correct, so alignment must be solved in advance; slow (gradual) allows mid-course adjustment.
• RSI is the core AGI-risk scenario because a fast compounding loop amplifies any misalignment (reward hacking, mesa-optimization) into an ever-more-powerful optimizer pursuing the wrong goal.
• The answer is bounded self-improvement: human in the loop, verify each step, cap the rate, provably-beneficial self-modification (Gödel / Darwin-Gödel machines).
• The near-term form is AI accelerating AI R&D, which frontier safety frameworks track as a key threshold.

Quick check

Next

RSI is the recursive, safety-framed version of self-improving agents; it amplifies mesa-optimization, and its near-term AI-R&D form is a key threshold in frontier safety frameworks. The inverted question — our obligations to the models themselves — is model welfare.