RLVR — reinforcement learning from verifiable rewards

RLHF has a component almost nobody defends on its merits: the reward model. You collect human preferences, fit a model to predict them, then optimise a policy against that model’s output. The policy is now being scored by a neural network with its own blind spots — and the whole point of RL is to search hard for inputs that maximise a score.

Given enough optimisation pressure, the policy finds the reward model’s blind spots. It always does. That is reward hacking, and it is the reason RLHF runs need a KL penalty holding the policy near its starting point: not because staying close is good, but because letting it wander lets it find the exploit.

Reinforcement learning from verifiable rewards removes the middleman.

What “verifiable” means

A reward is verifiable when a program can decide whether the output is correct, with no model in the loop:

• A maths answer that is compared against the ground truth after normalisation.
• Generated code that is run against a hidden test suite.
• A SQL query executed against a fixture database, its result set compared to the expected rows.
• A proof checked by a formal verifier such as Lean.
• A structured output validated against a JSON schema.

The common property is not “objective” — it is cheap, deterministic and un-gameable by argument. The grader does not read the reasoning and decide whether it seems convincing. It runs the thing.

This is also the honest limit of the method. “Write a good condolence letter” has no such grader, and pretending otherwise by asking a model to score it puts the hackable middleman right back in.

GRPO: the group is the baseline

Classic PPO needs a value network — a second model, roughly the size of the policy, trained to predict expected return so advantages can be computed. That is a lot of memory and one more thing to diverge.

Group relative policy optimisation makes an observation that is obvious in hindsight: if you sample G completions for the same prompt, the group’s own mean reward is a perfectly good baseline.

For a prompt with completions o₁…o_G and rewards r₁…r_G:

advantage(oᵢ) = (rᵢ − mean(r)) / std(r)

That is it. No value network, no bootstrapping. A completion that beat its siblings on the same prompt gets pushed up; one that did worse gets pushed down. Because the comparison is within a prompt, prompt difficulty cancels out — an easy prompt where everything scores 1 and a hard one where everything scores 0 both produce zero advantage and contribute no gradient.

The practical consequences:

• Memory roughly halves versus PPO, since the value network is gone.
• Prompts with no variance are wasted compute. If all G samples agree, the batch contributes nothing. Curriculum matters: you want prompts the model gets right sometimes.
• Group size is a real hyperparameter. Too small and the baseline is noisy; too large and you are spending generation budget on duplicates.

The four ways it gets gamed

“Un-gameable by argument” is not “un-gameable”. The grader becomes the specification, and the policy optimises the specification you wrote rather than the one you meant.

Passing the tests without solving the problem. If the test suite is visible or guessable, the model special-cases inputs. The fix is hidden tests, held out from training, plus tests generated independently of the solution.

Format gaming. If the checker extracts the final answer with a loose regex, the policy learns to emit several candidates and let the regex find a match. Make the extractor strict and reward exactly one well-formed answer.

Length exploitation. Where partial credit exists, longer outputs accumulate it. Either grade the final answer only, or penalise length explicitly.

Entropy collapse. This one is subtle and the most common in practice. The policy converges onto a single solution template that reliably passes, and diversity dies. Accuracy on the training distribution looks great; anything slightly different fails. Watch sample entropy and pass@k alongside pass@1 — if pass@1 climbs while pass@16 is flat, the model is not getting better, it is getting narrower.

Why it does not generalise everywhere

The gains from verifiable rewards concentrate where the grader is honest: competition maths, algorithmic code, formal proofs, structured extraction. These are the tasks where checking is genuinely easier than solving.

They transfer only partly to open-ended work. A model drilled on verifiable maths often does become a better reasoner in general — chains of thought get more disciplined — but the mechanism has nothing to say about tone, taste, or whether a recommendation is wise.

Which is why production post-training stacks are layered rather than either/or: supervised fine-tuning for the shape of the output, verifiable rewards where a grader exists, and preference-based methods for everything a program cannot check. Reaching for RLVR on a task whose reward you cannot actually compute is how you end up optimising a proxy again — the exact problem it was introduced to remove.

Check yourself

In one breath

Where a program can check the answer, let it: you lose the reward model and the exploit surface that comes with it. GRPO makes this cheap by using a group of samples as its own baseline. Then watch for the four gaming modes — hidden tests, strict extraction, length control, and entropy — because the grader is now your specification, and the policy will read it far more literally than you wrote it.