Test-time compute — buying accuracy at inference
You can spend more compute at inference instead of more parameters at training. Sampling many answers, revising one answer, and searching with a verifier are three different trades — and the one that works depends entirely on whether you can check the result.
Written and reviewed by Shreyash Prashu Last reviewed
What you'll learn
- Parallel vs sequential scaling — and why they fail on different problems
- Why a verifier, not the generator, is what sets the ceiling
- Best-of-N, self-consistency and revision, with the cost each actually carries
- How to allocate a budget when problems differ wildly in difficulty
Before you start
For most of the last decade, making a model better meant making it bigger, or training it longer. Test-time compute is the other axis: hold the weights fixed and spend more at inference — more samples, more revisions, more search.
It matters commercially because the trade is now explicit. A small model given a large inference budget can beat a large model answering once, on tasks with the right shape. Knowing which tasks have that shape is the whole skill.
Two ways to spend, and they are not interchangeable
Parallel scaling runs the same prompt N times independently and combines the results. Self-consistency takes the majority answer; best-of-N scores each with a verifier and takes the winner.
Sequential scaling runs one chain and keeps improving it: draft, critique, revise, repeat. Each step conditions on the last.
They fix different failures:
- Parallel fixes unlucky sampling. If the model knows the answer and lands on it 40% of the time, twenty samples will surface it.
- Sequential fixes a flawed approach. If the model’s first plan is wrong, resampling gives twenty variants of the same wrong plan. Revision can abandon it.
Which means parallel scaling helps when the model is capable but inconsistent, and does approximately nothing when it is confidently wrong — the case where it fails the same way every time.
The verifier sets the ceiling
Every one of these methods needs to answer the same question: of the candidates I now have, which is best? That selector, not the generator, is what determines how much your extra compute buys.
The options, in descending order of trustworthiness:
- A program. Unit tests, a formal checker, a database that either returns the right rows or does not. If you have this, best-of-N is close to free accuracy — generate more, keep what passes.
- Majority vote. No verifier needed, which is why self-consistency is so widely used. It assumes correct answers cluster and wrong ones scatter — reasonable for maths with a short final answer, useless for open-ended text where no two samples are identical.
- A learned verifier. A reward model scoring candidates, or a process reward model scoring each step. Strictly better than nothing and strictly worse than a program, with the same hackability problem any learned scorer has.
- The generator judging itself. The weakest option. A model that was confidently wrong once is usually confidently wrong about its own answer too. Self-critique helps most when the critique step gets information the draft did not have — an error message, a failed test, a retrieved document.
The single most useful question before building any of this: can I check the answer more cheaply than I can produce it? Where the answer is yes, test-time compute is extremely effective. Where it is no, you are paying N times the cost to pick with a coin.
What the curves actually look like
Accuracy against inference budget is not linear and not unbounded.
Early samples buy a lot. Going from one sample to four typically moves the needle hard, because you are mostly eliminating sampling noise. Beyond that, returns fall off sharply: by N=64 you are usually re-drawing from the same small set of answers the model finds plausible, and the correct one is either in that set or it never was.
This gives the practical shape of the trade. Doubling the budget repeatedly is a bad plan; matching the budget to the problem is a good one. Easy problems need one sample. Hard ones justify twenty. Spending twenty on everything wastes most of the budget on questions that were never in doubt.
The cheap way to approximate difficulty without a classifier: sample a few, and if they agree, stop. Disagreement among the first handful is a strong signal that more compute will help; unanimity is a strong signal it will not.
Cost, honestly
| Method | Extra generation cost | Needs a verifier | Latency shape |
|---|---|---|---|
| Single sample | 1× | No | One pass |
| Self-consistency (N) | N× | No | Parallel — latency ≈ one pass |
| Best-of-N + program | N× | Yes | Parallel, plus grading |
| Sequential revision (k steps) | ~k× | Helps a lot | Serial — latency ≈ k passes |
The row that surprises people is the last one. Parallel methods cost tokens but not wall-clock, because the samples run concurrently. Sequential revision costs both, since each step waits for the previous one. For an interactive product that difference usually decides the design before accuracy does.
Check yourself
Quick check
In one breath
Test-time compute trades inference cost for accuracy, but only along axes your problem actually has. Sample in parallel when the model is inconsistent, revise sequentially when its first approach is wrong, and be honest about the verifier — with a program you get close to free accuracy, with a model judging itself you mostly get a more expensive version of the same answer.