Training reasoning with verifiable rewards

At 3:07 a.m., a model is solving algebra problems for an automated grading service. There are no human reviewers available. Nobody wants to read 50,000 explanations to decide whether the answer to 7x - 13 = 85 is really 14.

But a small program can substitute 14 into the equation. It can return one point if the result is 85, and zero otherwise. The program does not get tired, prefer confident prose, or award extra credit for a dramatic “therefore.”

That changes the training problem.

Instead of asking a learned model to guess which answer a human would like, you can ask an executable checker whether the answer is correct. The checker’s result becomes the reward. This is the central idea behind verifiable rewards: using automatically checkable correctness as the optimisation signal for a language model.

The idea is simple. Making it useful is not.

From human preference to executable correctness

In ordinary reinforcement learning from human feedback (RLHF), people rank answers and a reward model learns to imitate those judgements. The language model is then trained against that learned proxy.

The proxy can capture clarity, tone, or appropriateness, but it is approximate. A policy can discover that polished formatting and confident wording earn a high score without improving the answer. It is optimising the judge, not necessarily the intended quality.

Verifiable rewards replace that learned judge with a program:

[ r(x,y) = C(x,y) ]

Here, x is the prompt, y is the completion, and C is a checker. The reward can be binary, such as pass or fail, or graded, such as the fraction of unit tests passed.

For 7x - 13 = 85, a checker evaluates the proposed value. For programming, it compiles code and runs hidden tests. For formal reasoning, a proof assistant verifies each inference.

The causal mechanism is:

1. The model samples a solution.
2. The checker evaluates it.
3. Training increases the probability of higher-reward solutions.
4. Repetition makes successful reasoning patterns more likely.

There is no learned estimate of human taste in the loop. That removes reward-model overfitting as one failure source, but not every way to optimise the wrong objective.

A worked example: eight attempts at one equation

Suppose the model receives:

Solve 7x - 13 = 85. Return the value of x on a line beginning with FINAL:.

The correct answer is 14. This narrow checker extracts the integer from the final non-empty line and verifies the equation. It does not judge the explanation.

import re

EXPECTED = 14

def reward(candidate: str) -> int:
    normalized = candidate.strip()
    if not normalized:
        return 0

    last_line = normalized.splitlines()[-1]
    match = re.fullmatch(r"FINAL:[ \t]*(-?\d+)", last_line, re.IGNORECASE)
    if match is None:
        return 0

    answer = int(match.group(1))
    return int(7 * answer - 13 == 85)

candidates = [
    "Subtract 13: 7x = 72\nFINAL: 14",
    "7x = 98\nFINAL: 14",
    "Divide both sides by 7\nFINAL: 13",
    "Add 13 to both sides, then divide by 7\nFINAL: 14",
    "Try x = 15\nFINAL: 15",
    "7(14) - 13 = 85\nFINAL: 14",
    "FINAL: fourteen",
    "The value satisfies the equation.\nFINAL: 14",
]

rewards = [reward(candidate) for candidate in candidates]

print("rewards:", rewards)
print("mean reward:", sum(rewards) / len(rewards))
print("accepted:", sum(rewards), "of", len(rewards))

It prints:

rewards: [1, 1, 0, 1, 0, 1, 0, 1]
mean reward: 0.625
accepted: 5 of 8

The first completion contains faulty algebra but receives one point because its final answer is correct. That is the consequence of an outcome reward: a score based only on the final result.

Using the group mean as a baseline, the mean reward is 0.625. Each successful attempt has relative advantage 0.375; each failed attempt has -0.625:

[ A_i = r_i - b ]

An advantage measures whether an action performed better or worse than the reference level. In simplified form, optimisation raises the log-probability of successful completions and lowers that of failed ones. Real algorithms add a reference policy, clipping, and token-level details, but the pressure is the same.

Sampling several solutions gives the optimiser alternatives for the same problem. Methods such as GRPO use this group comparison without a separate learned value model, although verifiable rewards can also work with other policy-optimisation methods.

Treat the checker as security-critical infrastructure: parse defensively, sandbox submitted code, add hidden and adversarial cases, and maintain a held-out evaluation set.

Why this is different from RLHF and DPO

RLHF asks, Which response would people prefer? DPO (Direct Preference Optimization) shifts probabilities using preferred and rejected responses. Its signal still comes from human or model-generated preferences, though it can use a fixed preference dataset without serving a reward model during optimisation. See DPO.

Verifiable-reward training asks, Did this completion satisfy an executable correctness condition?

That makes it strongest when:

• The desired property has a precise operational definition.
• Evaluation is cheaper than producing the answer.
• Passing the check correlates strongly with the real-world goal.

Competitive programming, formal proofs, and unambiguous mathematical answers often meet these conditions. Open-ended writing usually does not: there is no complete program for “insightful, accurate, humane, well-paced, and appropriate.” A word-count checker can produce the right length without producing a good essay.

An LLM-as-judge may help with writing, but it is again a learned reward model, with risks from bias, distribution shift, and optimisation against the judge. The approaches can be combined: preference training can improve style, while verifiable rewards improve correctness where correctness is executable.

Outcome rewards and process rewards

An outcome reward checks the final state: the equation balances, the program passes, or the proof is accepted.

A process reward scores intermediate steps. It can validate each algebraic transformation, proof rule, or program invariant. This provides denser feedback about where a long solution failed. A process checker might return 1, 1, 0, 0 for a four-step proof, rather than only a final pass or fail.

The price is formalisation. Defining valid intermediate states can be nearly as difficult as solving the task. Process rewards can also punish unusual but sound strategies if they recognise only routes seen in their examples. Outcome rewards are sparse, but leave more freedom for discovery.

The choice need not be binary. A code task might combine hidden-test performance, compilation, static checks, and resource limits. Those weights are part of the specification and must be validated against the actual task.

Why longer chains of thought can appear

Longer solution traces need not be written into the objective. If a short guess receives zero while a 40-token trace that factors an expression, catches a sign error, and corrects it receives one, optimisation raises the probability of the successful trajectory. Repeated rewards for decomposition, checking, backtracking, or tool use can make those behaviours common.

Longer reasoning is therefore an optimisation outcome. It can add useful computation, but also add opportunities for error. Without a length penalty, the model may learn bloated traces; with a tight context limit, successful reasoning may be truncated. A final-answer checker can also reward elaborate explanations that are no more reliable.

At deployment, the counterpart is test-time compute: spending more computation on a prompt rather than only during weight updates. For example, sample eight candidates, check each, and return an accepted one. If each independent attempt succeeds with probability 0.6:

[ 1 - (1 - 0.6)^8 = 1 - 0.4^8 = 0.99934464 ]

That is about 99.93 percent, but only under genuine independence and a reliable checker. Models’ mistakes are often correlated, and eight attempts cost roughly eight times the generation work.

Use one completion for easy requests and escalate to multiple checked samples when the value or uncertainty justifies the latency and cost.

The risks move into the checker

Replacing a learned reward model removes one class of proxy failure, not the need for safety.

Reward hacking targets gaps in the checker: code can hard-code visible tests, a parser can accept an unintended value, or a proof checker can rely on an accidentally permissive axiom. This is still Goodhart’s law; the gap has moved into code, test coverage, or the execution environment. See reward hacking.

Use independently generated hidden tests, known-good and known-bad checker tests, varied inputs, sandboxing, static analysis, resource limits, and human review of high-stakes specifications.

A narrow objective can also improve theorem acceptance while harming adjacent abilities such as explaining uncertainty or handling tasks without a verifier. Finally, a binary checker trains acceptance, not honest uncertainty.

Calibration means stated confidence matches observed correctness. A model that says it is 80 percent confident across 100 answers should be correct about 80 times. High verified pass rates do not establish calibration, especially when the checker disappears on novel deployment tasks. Measure calibration on easy, hard, in-distribution, and out-of-distribution tasks, and train or tune abstention where appropriate.

Failure modes you will see first

First symptom	Likely cause	Fix
Training pass rate jumps, but independent tests remain poor	Parser or visible-test loophole	Fuzz the checker, add hidden tests, sandbox execution, and inspect accepted outputs
Reward stays near zero and outputs are malformed	Sparse signal or unclear contract	Add format examples, curriculum tasks, and partial or process checks
Held-out accuracy falls	Distribution or visible-test overfitting	Generate fresh instances and use an independently maintained checker
Unusual valid solutions disappear	Process checker recognises only familiar routes	Relax it, emphasise final correctness, and test multiple valid paths
Confidence rises without hard-task improvement	Acceptance was optimised without calibration	Add calibration and abstention evaluations

What to remember

• Verifiable rewards sample an answer, run a checker, and increase the probability of answers that pass.
• They fit math, code, formal proofs, and other tasks with executable specifications better than open-ended writing.
• Outcome rewards provide freedom but sparse credit; process rewards provide denser credit but require a trustworthy definition of valid work.
• Longer reasoning can emerge when extra computation improves the chance of passing, but test-time search has real cost.
• A checker is not an oracle. Validate it independently and measure calibration separately.

Quick check

0/3

Q1

A verifiable reward comes from an executable correctness check, while an RLHF reward comes from a learned model of human preferencesA verifiable reward always uses more tokens than RLHFRLHF can train only code modelsA verifiable reward never needs evaluation data

Check answer

Q2

The mean reward is 0.25, so every solution has a negative advantageThe mean reward is 0.625; passing solutions have positive relative advantage and failing solutions have negative relative advantageThe mean reward is 1, because at least one solution passedThe checker cannot provide a training signal unless all eight solutions pass

Check answer

Q3

Verifiable rewards based only on email lengthA formal proof checker for every sentenceRLHF or DPO using reliable preference data, possibly combined with narrower automatic checksReject every answer that is not exactly identical to a reference email

Check answer