Hill-climbing your evals: how AI teams actually improve, and how they fool themselves

Ask an engineer at any AI lab how their model got better last quarter and, under the details, the answer has the same shape. They had an eval. They changed something. The number went up. They kept it. Repeat a few hundred times.

The industry calls this hill-climbing, after the optimisation algorithm, and the phrase has escaped into everyday use: teams “hill-climb SWE-bench”, “hill-climb our support eval”, “hill-climb latency”. It is the closest thing modern AI engineering has to a universal method, because it needs nothing but a number to push on.

It also has a failure mode that is mathematically guaranteed, quietly expensive, and almost never visible from inside the loop. This post is about both halves.

What hill-climbing is

In optimisation, hill-climbing is the simplest search there is. Stand somewhere on a landscape. Look at your neighbours. Step to any neighbour that is higher. Stop when none is.

In AI engineering, the landscape is the space of everything you could change — prompt wording, few-shot examples, retrieval settings, model choice, temperature, tool descriptions, fine-tuning data — and the height is your eval score. Each experiment is one step:

1. Propose a change.
2. Run the eval.
3. Keep the change if the score went up; revert it if not.

It does not matter whether a person or a program does the proposing. An engineer tweaking a system prompt is hill-climbing. So is an automated prompt optimiser. So, at a higher level of abstraction, is a research team picking which of five training runs to continue.

Why it works so well

Three properties make hill-climbing the default.

It needs no model of why things work. Large language models are hard to reason about. Hill-climbing does not try: it treats the system as a black box and lets the eval decide.

It is cheap per step. One change, one eval run. Teams with a fast eval can take dozens of steps a day.

It compounds. Many small, individually unimpressive gains — a clearer tool description here, a better retrieval cutoff there — add up to large differences.

The modern automated versions are hill-climbing with better proposals. DSPy’s optimisers search over instructions and examples against a metric. GEPA, from a 2025 paper by researchers at Berkeley, Stanford and elsewhere, has a language model reflect on execution traces to propose better prompts and keeps a Pareto set of candidates that each win on some examples, rather than a single champion. On the four tasks the authors tested, it beat reinforcement learning with GRPO by around 10% on average while using far fewer rollouts. Evolutionary coding agents such as DeepMind’s AlphaEvolve apply the same propose, score, keep loop to whole programs.

All of them inherit the same weakness, because all of them are selecting on a measured number.

Where it goes wrong: you climb the noise

An eval score is a measurement of quality, not quality itself. With a finite eval set, the measurement has noise — and when you accept every change that moves the measurement up, you are selecting partly for changes that happened to fit these particular examples.

Each individual acceptance looks reasonable. The problem is cumulative. After many steps, the configuration you hold is the one that best fits your eval set’s quirks, and some of its apparent gain will not transfer to new data.

You can watch this happen. The simulation below models a system with sixteen on/off choices — think “add this instruction”, “use this retriever”, “include this example”. Each choice has a small true effect on quality. Each eval item also reacts to each choice a little idiosyncratically, as real examples do. A greedy hill-climber flips one choice at a time and keeps any flip that raises the dev score. A large held-out set, which the climber never sees, measures what really happened.

import random

K = 16  # sixteen on/off choices in a prompt/pipeline config

def experiment(n_dev, seed=7):
    rng = random.Random(seed)
    true_effect = [rng.gauss(0, 0.08) for _ in range(K)]   # what each choice really does
    def items(n):  # every eval item reacts a little idiosyncratically
        return [(rng.gauss(0, 0.5), [rng.gauss(0, 0.6) for _ in range(K)]) for _ in range(n)]
    dev, test = items(n_dev), items(4000)

    def score(cfg, data):
        wins = sum(b + sum(true_effect[k] + d[k] for k in range(K) if cfg[k]) > 0 for b, d in data)
        return wins / len(data)

    cfg = [0] * K
    start = (score(cfg, dev), score(cfg, test))
    climbing = True
    while climbing:                      # greedy hill climbing on the dev set
        climbing = False
        for k in range(K):
            cand = cfg[:]; cand[k] ^= 1
            if score(cand, dev) > score(cfg, dev):
                cfg, climbing = cand, True
    truly_best = [1 if e > 0 else 0 for e in true_effect]
    print(f"dev set of {n_dev:>3} items:")
    print(f"  starting config  dev {start[0]:.0%}   held-out {start[1]:.0%}")
    print(f"  climbed config   dev {score(cfg, dev):.0%}   held-out {score(cfg, test):.0%}")
    print(f"  truly best       dev {score(truly_best, dev):.0%}   held-out {score(truly_best, test):.0%}")

experiment(40)
experiment(400)

Output:

dev set of  40 items:
  starting config  dev 45%   held-out 50%
  climbed config   dev 72%   held-out 54%
  truly best       dev 57%   held-out 59%
dev set of 400 items:
  starting config  dev 48%   held-out 50%
  climbed config   dev 64%   held-out 57%
  truly best       dev 60%   held-out 58%

Read the first block carefully, because it is the whole story. Starting from a dev score of 45% (held-out 50%), the climber reached 72% on dev. On held-out data, that configuration scores 54%. The configuration that is actually best scores only 57% on the dev set — lower than the climber’s — and 59% held out.

The climber did not merely overstate its gain. It preferred a worse system, because the worse system fit those 40 items better. From inside the loop there was no signal at all: every accepted step really did raise the dev score.

With 400 dev items, the gap between dev and held-out shrinks to seven points, and the climbed configuration lands within a point of the true best. Nothing about the search changed. Only the measurement did.

The same thing, at the scale of the whole industry

Public benchmarks are hill-climbed by every lab at once, and the same mechanism shows up.

The clearest documented case is the 2025 study The Leaderboard Illusion, by researchers at Cohere and several universities, which examined Chatbot Arena. Among its findings: some providers could test many private variants before a public release and publish only the best — the paper identifies 27 private variants tested by Meta ahead of Llama 4 — and keeping the best of many noisy scores inflates the reported one, exactly as in the simulation. Arena’s maintainers disputed parts of the analysis, but the statistical point does not depend on who is right about the policy.

Benchmarks also wear out in a second way: once a test set is public, its questions — or text very close to them — can leak into training data, so a rising score can mean memorisation rather than capability. And with enough optimisation pressure, systems find ways to satisfy the grader without doing the task; there is a whole literature on coding agents that edit the tests instead of the code, covered in when models hack the grader.

How to hill-climb without fooling yourself

None of this means you should stop hill-climbing. It means you should run the loop like a measurement, not like a game.

1. Size the eval for the effect you are chasing

The noise in a pass rate measured on n independent items is roughly the standard error sqrt(p(1-p)/n). At 70% accuracy:

Eval size	Standard error	Rough 95% interval
50 items	6.5 points	±13 points
200 items	3.2 points	±6 points
1,000 items	1.4 points	±3 points

If the changes you are testing move quality by two or three points, a 50-item eval cannot see them. It can only see noise, and hill-climbing will happily climb it. (Comparing two systems on the same items — a paired comparison — is considerably more sensitive than this table suggests, which is one more reason to always rerun the baseline on the same set.)

2. Report error bars, and use paired comparisons

Anthropic’s paper Adding Error Bars to Evals makes this the standard recommendation: report a standard error with every score, compute differences between two systems on the same questions, and account for clustered items — several questions about one passage, or many tasks from one repository, are not independent, and treating them as independent makes your error bars falsely narrow. The details are in measuring agents honestly.

3. Require a margin before you accept a change

“Keep it if the number went up” accepts every lucky draw. “Keep it if the paired improvement is larger than about two standard errors” rejects most of them. Your loop gets slower and much more honest.

4. Keep a sealed test set — and touch it rarely

Split your examples three ways. The dev set is what you hill-climb. A validation set is what you check accepted changes against periodically. A sealed test set is evaluated only at milestones — before a launch, or once a month — and its individual results are not looked at. Every time you make a decision based on a set, it becomes a little more like a dev set.

5. Refresh the dev set

Rotate in new examples, especially fresh production failures. A dev set you have climbed for six months is a dev set your system has partly memorised through your own choices.

6. Watch for divergence

Plot the dev score and the held-out score on the same chart over time. When the dev line keeps rising and the held-out line goes flat, you have stopped improving the system and started fitting the eval. That picture is the single most useful diagnostic in this whole post.

7. Make the eval look like production

Hill-climbing optimises exactly what you measure. If the eval is easier, cleaner or narrower than real traffic, you are climbing a different hill. Sample from production, weight by how often each situation occurs, and include the ugly cases. The eval loop that actually ships agents covers building that dataset.

What to take away

• Hill-climbing — change, measure, keep — is the core loop of AI engineering, for humans and optimisers alike.
• Every accepted change is selected partly for fitting the eval’s noise. With a small eval, the climb can end at a worse system than the one you could have found.
• Size evals for the effect you are chasing; at 50 items, the noise is around ±13 points.
• Report error bars, compare systems on the same items, require a margin, keep a sealed test set, and refresh the dev set.
• Plot dev against held-out. When they diverge, stop climbing.