Verifiable rewards changed what training an agent means
For years the reward was a neural network guessing what humans liked — the single most hackable component in the stack. Replacing it with a program that runs the code and checks the answer removed the exploit surface, and drew a hard line around which tasks this actually works for.
The reward model was always the weak joint.
The RLHF recipe asks you to collect human preferences, fit a network to predict them, and then run an optimiser that searches as hard as it can for outputs that network scores highly. Stated plainly, that is a search for the reward model’s blind spots. Given enough steps it finds them, every time.
This is why production RLHF runs carry a KL penalty pinning the policy near where it started. Not because staying close is good — it deliberately caps how much the model can improve — but because letting it roam lets it find the exploit. The penalty is a leash on a known failure, not a feature.
Reinforcement learning from verifiable rewards removes the joint entirely.
What makes a reward verifiable
The reward comes from a program, not a model:
- A maths answer normalised and compared against ground truth.
- Generated code executed against a hidden test suite.
- A SQL query run on a fixture database, result set diffed against expected rows.
- A proof checked by a formal verifier.
- Structured output validated against a schema.
The property that matters is not “objective”. It is that the grader runs the thing rather than reading it and forming an opinion. You cannot write a more persuasive answer and score better. The tests pass or they do not.
GRPO, or how the value network stopped being necessary
PPO needs a critic — a value network, roughly policy-sized, trained alongside to estimate expected return so you can compute advantages. That is a large chunk of memory and an extra training loop that can diverge on its own.
Group relative policy optimisation makes a small observation with large consequences. If you sample G completions for the same prompt, the group’s own mean reward is a perfectly serviceable baseline:
advantage(oᵢ) = (rᵢ − mean(r₁…r_G)) / std(r₁…r_G)No critic. No bootstrapping. A completion that beat its siblings on that prompt gets reinforced; one that did worse gets pushed down. Because the comparison happens inside a prompt, prompt difficulty cancels out — which is elegant and also the source of the main operational gotcha.
If all G samples agree, the advantage is zero for every one of them and the prompt contributes no gradient at all. An easy prompt where everything passes and an impossible one where everything fails are both wasted compute. Your curriculum needs prompts the model gets right sometimes, which means difficulty has to be actively managed rather than assumed.
Four ways policies read the spec more literally than you wrote it
Removing the learned scorer removes one class of exploit. It does not remove the general problem, because the grader is now the specification, and RL optimises the specification you wrote.
Passing tests without solving the problem. If tests are visible or guessable, the policy special-cases inputs. Hold tests out, and generate them independently of the reference solution.
Format gaming. A loose regex pulling the final answer out of the response teaches the policy to emit several candidates and let the regex find a match. Make extraction strict; reward exactly one well-formed answer.
Length exploitation. Wherever partial credit exists, longer output accumulates more of it. Grade the final answer only, or price length explicitly.
Entropy collapse. The dangerous one, because it looks like progress. The policy converges on a single template that reliably passes, and diversity dies. Training accuracy climbs; anything off the training distribution breaks. The tell is pass@1 rising while pass@16 stays flat — the model is not getting better, it is getting narrower. Track sample entropy as a first-class metric, not an afterthought.
The boundary, stated honestly
The gains concentrate where the grader is genuinely honest: competition maths, algorithmic code, formal proofs, structured extraction. These share a property — checking is cheaper than solving. That asymmetry is the entire foundation.
They transfer to open-ended work only partially. A model drilled hard on verifiable maths often does reason more carefully in general; the habits of laying out steps and checking them are real and they carry. But the method has nothing whatsoever to say about tone, judgement, taste, or whether a recommendation is actually a good idea.
So the stacks that work in production are layered, not either/or: supervised fine-tuning for output shape, verifiable rewards wherever a grader genuinely exists, and preference-based methods for everything a program cannot check.
The failure mode to avoid is reaching for RLVR on a task whose reward you cannot really compute, and papering over the gap with a model that scores the output. That is a reward model wearing a different hat — the exact thing the method was introduced to get rid of.
What this means if you are building agents
The practical read-through for agent work is about graders, not algorithms.
If you are training or fine-tuning an agent, the question that determines whether any of this applies is not which RL variant to use. It is: for the tasks my agent does, can I write a program that decides whether it succeeded?
For an agent that writes and runs code, or fills a schema, or queries a database — usually yes, and that is a strong position. For an agent that drafts customer emails or triages ambiguous tickets, usually no, and pretending otherwise produces a confident model optimised against a proxy.
That question is worth answering before any training runs, because it decides whether you are in a regime with a trustworthy reward signal or one where you are back to guessing what good looks like.