Reasoning models & test-time compute

It is 3 a.m. Your database migration has failed halfway through. The new service expects a column that the old service still writes, two foreign keys are in the wrong order, and the rollback script was never tested.

A standard language model can produce a polished migration plan in two seconds. It can also miss the foreign-key problem while confidently recommending the rollback script. The answer sounds finished because the prose is finished.

Making the model larger is one way to improve it, but it is expensive and slow. Another approach is to give the model more computation when it needs it: let it work through constraints, inspect candidate solutions, find contradictions, and revise before answering.

A reasoning model is trained or post-trained to use additional or adaptive inference-time computation before returning an answer—often a longer hidden reasoning trajectory, and sometimes sampling, search, or verification. Here, test-time compute means the extra per-query budget beyond an ordinary one-pass response. OpenAI’s o-series, DeepSeek-R1, and related models made this behavior a mainstream product feature. The important change is where the budget is spent.

Training once, thinking per request

An LLM has two different compute bills.

Training compute builds the model. It processes datasets and adjusts weights, the numerical parameters that store learned behavior. That bill is paid before your request and can improve knowledge, representations, and general capability.

Inference compute runs the model after you send a request. It generates tokens, examines possibilities, calls tools, or searches for a better answer. You pay this bill again for every request. In this lesson, test-time compute is the additional or adaptive inference beyond an ordinary one-pass response: hidden reasoning tokens, multiple samples, search, or verification.

A standard chat call typically begins generating the user-visible answer. A reasoning model may first generate hidden or separately accounted-for reasoning tokens. This is a behavior and serving convention, not a set of separate neural modules.

For the migration incident, it might:

1. list schema and compatibility constraints;
2. propose a migration order;
3. check whether old and new services can coexist;
4. find the foreign-key conflict and revise the plan with a rollback checkpoint.

These are a useful mental model, not a promise of four neat internal modules. The actual process remains neural text generation. Some systems also use additional sampling or search.

Generation is sequential. If a request contains 800 input tokens and produces a 300-token answer, a standard model generates roughly 300 output tokens. A reasoning model might generate 2,400 hidden reasoning tokens and then the same 300-token answer. The reader sees 300 tokens in both cases, but the serving system may process 3,500 tokens in the second case.

More thinking usually means more latency and consumes GPU capacity. Providers also differ in whether they expose or bill hidden reasoning, show a summary, or offer an effort setting instead of a token number. Check the current API documentation and invoice; visible answer length is not enough to estimate cost.

What extra compute is buying you

The mechanism is more specific than “the model thinks harder.” Extra computation gives the model more chances to construct, compare, and reject possible solutions.

A reasoning-oriented model is trained on examples and rewards that favor successful multi-step solutions. Reinforcement learning might reward a correct answer, a passing test suite, or a valid proof. DeepSeek-R1’s use of GRPO, or Group Relative Policy Optimization, is one example; other systems use supervised fine-tuning, reinforcement learning, verifiers, or distillation.

Training teaches the model how to spend extra tokens. Inference supplies them.

This is why a smaller model with enough useful test-time work can beat a larger model on a narrow hard task. It is not universal: a larger model may still win on broad knowledge, language quality, multimodal understanding, or tasks where the smaller model’s reasoning is wrong.

For the migration, extra thought can improve ordering and compatibility reasoning. It cannot discover an undocumented production dependency absent from the prompt, or prove a rollback script works without running it. Reasoning is not a substitute for data, tools, or tests.

The budget has a sweet spot

Reasoning effort is a compute budget, not a magic quality slider. More budget gives the model room to work, but the value of each additional token eventually falls.

A typical hard-task curve of accuracy versus allocated reasoning tokens looks like this:

import math

def accuracy(budget):
    rise = 0.42 + 0.46 * (1 - math.exp(-budget / 900))
    overthink = max(0.0, (budget - 3000) / 5000) * 0.11
    return min(0.9, rise - overthink)

print(f"{'budget':>7} {'accuracy':>9} {'relative cost':>14}")
for budget in [0, 500, 1500, 3000, 5000, 8000]:
    print(f"{budget:7d} {accuracy(budget) * 100:8.0f}% {budget / 1000:13.1f}")

budgets = range(0, 8001, 100)
best = max(budgets, key=accuracy)
print(f"\npeak accuracy at ~{best} thinking tokens — past that, you pay more for less")

At first, more work fixes obvious mistakes. Then returns diminish. Eventually the model may reopen a correct decision, invent a new interpretation, or add complexity with more opportunities to fail. This is the overthinking regime. Increasing a maximum may also do nothing if the model stops early; it can cause truncation rather than overthinking if the limit is too low.

Thinking budget, effort, and output length

Providers expose reasoning control as:

• a numeric maximum for reasoning or hidden tokens;
• a setting such as low, medium, or high effort;
• a model variant for faster or deeper reasoning;
• a general output limit that may include reasoning, visible output, or both.

These controls are not interchangeable. A budget is an upper bound, not a command to use every token. A visible-answer limit is not necessarily a thinking limit, and a context window is not a promise that all its space will be used for reasoning.

The migration numbers show the trade-off. Suppose a normal request uses 1,050 billed tokens—800 input and 250 output—while a reasoning request uses 3,500: 800 input, 2,400 hidden reasoning, and 300 visible output. The reasoning version uses about 3.3 times as many billed tokens. At an assumed rate of $10 per million billed tokens, 100,000 requests cost about $1,050 versus $3,500 before input/output price differences, caching, or discounts.

If reasoning raises success from 82% to 91% on a costly migration, that may be a bargain. If it raises simple extraction from 98.1% to 98.2%, it is probably not.

Do not prompt it like a standard model

You may know chain-of-thought prompting: asking a standard model to “think step by step.” That can help a model that would otherwise answer too quickly.

A reasoning model is trained to perform that internal work already. Give it a precise task instead:

• state the desired outcome and constraints;
• provide the schema, evidence, and rollback requirements;
• identify facts that must be checked with tools;
• request a concise plan with assumptions and verification steps;
• use a schema when software will consume the answer.

The internal reasoning may be hidden, summarized, or unavailable. A long explanation is not necessarily a faithful transcript of the computation.

Ask for useful artifacts—assumptions, a short justification, citations, test results, or a compact decision record—rather than demanding a private scratchpad.

Choosing the right tool

The decisive question is not “Is reasoning better?” but “Where is the bottleneck?”

For the migration service, a production design might:

1. route routine requests to a fast model;
2. retrieve the runbook and inspect the current schema for grounded requests;
3. send complex plans to a reasoning model with a fixed budget;
4. validate ordering and rollback steps, with human approval before production changes.

This is a model routing problem, not a contest to use the most expensive model everywhere. Route using task type, tool state, complexity, and validation failures—not only the model’s confidence.

Use LLM evals with easy, medium, and genuinely hard examples. At each effort level, measure correctness, format validity, total and tail latency, billed tokens, and human correction or escalation rates. Choose the smallest budget that meets a concrete target, such as “90% correct on the hard migration set, p95 below eight seconds, and under four cents per request.”

Failure modes you will actually see

The response stops halfway through the plan. A truncated answer, unfinished JSON object, or missing rollback step usually means the reasoning or total output limit was too small. Raise the relevant limit, shorten the requested artifact, or split planning from execution.

The answer gets worse at higher effort. A correct calculation may be replaced by an elaborate incorrect one, or a plan may gain needless steps. Compare budgets on a fixed evaluation set, cap the range, and add an independent checker. Route simple tasks to a faster model.

The model reasons from a false premise. A migration plan may mention a nonexistent table, or a support agent may cite an obsolete policy. Add RAG or an authoritative tool, require source identifiers, and validate retrieved facts. More scratchpad tokens do not make stale information current.

The output is logically sound but unusable. Use schemas, constrained output where appropriate, and a parser that rejects invalid responses.

Production latency and confidence mislead you. Mean latency hides long-tail requests that consume the full budget, while a careful-sounding explanation is not proof. Track p95/p99 latency and token usage; run tests, execute calculations, validate JSON, compare evidence, and require approval for irreversible actions.

In one breath

• A reasoning model is trained or post-trained to use additional or adaptive inference-time computation before its final answer; test-time compute is the extra per-query budget beyond an ordinary one-pass response.
• Training compute changes the model once; inference compute is spent again for every request.
• Extra reasoning helps when decomposition, search, and checking attack the task’s real difficulty. It cannot supply missing facts or replace tools.
• Accuracy usually rises with diminishing returns, then may plateau or decline as cost and latency continue rising.
• A reasoning budget is a cost-and-latency control, not a quality guarantee. Tune it on an evaluation set.
• Give clear objectives, constraints, evidence, and output requirements. Route only genuinely difficult work to the expensive path.

Quick check

Next

Model routing sends only hard queries to expensive models. LLM evals tests whether extra thinking improves correctness. For current or private facts, start with RAG basics and add reasoning after retrieving the evidence.