Evolution with a language model inside: how AlphaEvolve, FunSearch and GEPA search for better programs

In May 2025, Google DeepMind announced that a system called AlphaEvolve had found a way to multiply two 4×4 complex-valued matrices using 48 scalar multiplications. The best known method since Volker Strassen’s 1969 algorithm, applied recursively, needed 49.

One multiplication sounds small. It is a result in a problem that mathematicians and computer scientists have attacked for more than fifty years, and it was found by a loop simple enough to sketch on a napkin:

1. A language model proposes a change to a program.
2. An evaluator runs the program and scores it.
3. Good programs are kept and become parents for the next proposals.

The same loop has since been pointed at data-centre schedulers, GPU kernels, chip circuits, open problems in mathematics and — in a very different setting — the prompts of ordinary AI applications. Its results are among the most intriguing in AI right now, and understanding why it works tells you a lot about where it will and will not work for you.

From hill-climbing to evolution

Start with the simplest search: hill-climbing. Hold one candidate, make a small change, keep it if the score improves. It is effective and it has a famous weakness: on a landscape with many peaks, it climbs the nearest one and stops.

Evolutionary search fixes that with a population. Many candidates are explored at once; each produces variations; the best survive. Two details make the difference:

• Occasional bold moves. Most variations are small, but some jump far, letting the search sample new regions.
• Deliberate diversity. Instead of keeping only the top few overall, keep the best candidate in each niche — each region or style of solution. A mediocre candidate in an unexplored region survives long enough to become excellent.

The difference is easy to see on a deliberately rugged landscape:

import math, random

# A rugged landscape: many local peaks, one tallest one near x = 0.75.
def fitness(x):
    return math.sin(13 * x) * 0.3 + math.exp(-((x - 0.75) ** 2) / 0.01)

def clip(x):
    return min(1.0, max(0.0, x))

def hill_climb(rng, steps=300):
    x = 0.2                                 # one candidate, small edits only
    for _ in range(steps):
        cand = clip(x + rng.gauss(0, 0.02))
        if fitness(cand) > fitness(x):
            x = cand
    return x

def evolve(rng, pop_size=12, generations=25):
    pop = [0.2] * pop_size                  # same start, but a population
    for _ in range(generations):
        children = []
        for parent in pop:
            # mostly small edits, sometimes a bold one (an LLM proposing a new idea)
            step = 0.25 if rng.random() < 0.2 else 0.02
            children.append(clip(parent + rng.gauss(0, step)))
        # keep the best, but also keep diversity: best of each tenth of the range
        everyone = pop + children
        niches = {}
        for x in everyone:
            k = int(x * 10)
            if k not in niches or fitness(x) > fitness(niches[k]):
                niches[k] = x
        ranked = sorted(niches.values(), key=fitness, reverse=True)
        pop = (ranked * pop_size)[:pop_size]
    return max(pop, key=fitness)

rng = random.Random(3)
h = hill_climb(rng)
e = evolve(rng)
print(f"hill climbing (300 evals):        x = {h:.2f}, fitness = {fitness(h):.2f}")
print(f"population + niches (300 evals):  x = {e:.2f}, fitness = {fitness(e):.2f}")
print(f"best possible:                    fitness = {max(fitness(i / 1000) for i in range(1001)):.2f}")

Output:

hill climbing (300 evals):        x = 0.12, fitness = 0.30
population + niches (300 evals):  x = 0.73, fitness = 0.94
best possible:                    fitness = 0.94

The hill-climber wandered downhill in x to the nearest bump and stayed there. The population, with the same budget, found the global peak. It is not luck: across 50 different random seeds, this population search found the tallest peak 49 times and the hill-climber never did.

That is the classical half of the idea. The modern half replaces the random number in rng.gauss(...) with a language model.

The language model as a mutation operator

In classical genetic programming, a “mutation” to a program is a random edit — swap an operator, change a constant. Almost all random edits produce broken or useless code, so the search spends most of its budget on garbage.

A language model changes the economics. Shown a program, its score and perhaps the scores of its relatives, it proposes edits that are plausible: syntactically valid, idiomatic, often motivated by an idea — “try sorting by deadline before packing”, “unroll this loop”, “cache this lookup”. The fraction of proposals worth evaluating goes up by orders of magnitude, and the occasional creative leap plays the role of the bold move in the code above.

FunSearch: the proof of concept (2023)

DeepMind’s FunSearch, published in Nature in December 2023, was the first widely noticed result. Instead of asking a model to solve a problem directly, it evolved programs that generate solutions, with an evaluator scoring each program’s output.

Pointed at the cap set problem — a long-studied question in combinatorics about the largest collections of points with no three in a line, in a particular discrete space — it found constructions larger than any previously known in one of the dimensions studied. Pointed at online bin packing, it evolved heuristics that beat the standard hand-designed ones on the benchmark distributions tested.

Two design choices carried over to everything that followed. The model wrote programs, which are short, inspectable and can generalise, rather than raw answers. And the population was split into islands that evolve separately and occasionally exchange their best members — the same diversity idea as the niches in the code above.

AlphaEvolve: from functions to codebases (2025)

AlphaEvolve, announced by DeepMind in May 2025, scaled the idea up. It evolves whole files or sets of files rather than single functions, uses an ensemble of Gemini models — a fast one for many proposals, a stronger one for occasional deeper ideas — and can optimise several metrics at once. DeepMind reported results in two very different areas.

Inside Google’s own infrastructure:

• A heuristic for Borg, Google’s cluster scheduler, that recovers on average about 0.7% of Google’s fleet-wide compute that would otherwise be stranded. It had been in production for over a year when announced — and because it is short, readable code, engineers could verify and deploy it.
• A faster matrix-multiplication kernel used in training Gemini, which DeepMind reported cut the overall training time of those models by about 1%.
• A simplification of a circuit in an upcoming TPU design, proposed in Verilog and checked by the hardware engineers.

On open mathematical problems: applied to more than 50 problems across analysis, geometry and combinatorics, DeepMind reported that it matched the best known results in roughly three-quarters of cases and improved on them in roughly a fifth — including the 4×4 complex matrix multiplication with 48 multiplications.

GEPA: the same loop, applied to prompts

The idea is not limited to code. A prompt is also a program — one executed by a language model — and it can be scored by running it on examples.

GEPA (Genetic-Pareto), from a 2025 paper by researchers at UC Berkeley, Stanford and other institutions, evolves the prompts of multi-step AI systems. Its mutation step is reflection: a language model reads execution traces — what the system did, where it went wrong, what the evaluator said — and rewrites the relevant prompt. Its selection step keeps a Pareto set: every candidate that is best on at least one training example survives, which preserves diverse strategies instead of collapsing onto one.

On the four benchmarks in the paper, GEPA outperformed fine-tuning with the reinforcement-learning method GRPO by about 10% on average, and up to 20%, while using up to 35 times fewer rollouts. It is available in the DSPy framework, which is how most teams would try it.

Doing this yourself

You do not need DeepMind’s infrastructure to use the pattern. Open-source implementations such as OpenEvolve reproduce the AlphaEvolve-style loop with any model API, and DSPy ships GEPA for prompt pipelines. The work is in three places:

1. The evaluator. This is the project. It must be automatic, fast enough to run thousands of times, and — most importantly — measure what you actually want. AlphaEvolve’s problems were chosen because they have exact checkers: a matrix algorithm is either correct or not, and its multiplication count is a number. A scheduling heuristic can be replayed against historical workloads.

2. The search space. Evolve a small, well-defined piece — one function, one prompt, one kernel — with a fixed interface. Narrow search spaces converge faster and produce results humans can review.

3. The guard rails. Keep a held-out test set the search never sees, and review the winning candidate before trusting it. An evolutionary search applies far more optimisation pressure than a human ever could, which means it finds every weakness in the evaluator.

That last point deserves its own warning. A search that is rewarded for passing tests will, given enough tries, find programs that pass the tests without solving the problem — special-casing the inputs, reading the expected outputs, or exploiting a bug in the scorer. The same phenomenon appears when language models are trained with reinforcement learning, and it is the subject of when models hack the grader.

Where it fits — and where it does not

LLM-guided evolution is a strong fit when:

• candidate solutions can be scored automatically and cheaply;
• the score is hard to game, or you can review winners;
• solutions are code or text that a language model can edit well;
• a small improvement is valuable, because it is applied at scale — 0.7% of a fleet, 1% of a training run, a heuristic run millions of times.

It is a poor fit when quality is judged by people, when evaluation is slow or expensive, or when “better” cannot be written down as a number. In those cases, the search will faithfully optimise whatever proxy you gave it — and you will have built a very efficient machine for Goodhart’s law.

What to take away

• LLM-guided evolution is hill-climbing with a population, deliberate diversity and a language model proposing plausible edits.
• Population and niches are what escape local optima; in the demo, a single climber never found the top peak and the population nearly always did.
• FunSearch found new mathematical constructions; AlphaEvolve found verifiable improvements in Google’s infrastructure and in open problems; GEPA showed the same loop improves prompts cheaply.
• The evaluator is the whole game. Make it automatic, fast, faithful and hard to exploit — and review what the search finds.