How reasoning distillation gives small models stronger math and code behavior locally

At 3:07 a.m., a support engineer wants a local assistant to check a retry calculation. The laptop has 16 GB of memory. A small language model answers quickly, but it says 36. That is the number of failed requests, not the number of total attempts.

A larger reasoning model gets the sequence right:

• 1,200 initial attempts
• 3 percent of 1,200 equals 36 failures
• 36 retries
• 1,236 total attempts

The larger model is better at this kind of decomposition, but sending every question to it is slow, expensive, or impossible when the data must stay on the laptop.

The useful lesson is not that a small model magically becomes OpenAI o3. It is that parameter count is only one way to buy capability. You can also teach a smaller model a useful problem-solving routine by showing it many examples of a stronger model working through problems.

That is reasoning distillation, and it is one of the most practical ways to move frontier-style reasoning out of the data center.

The headline needs a footnote

“o3-level reasoning on your laptop” is a fair description only if you attach three words: on selected tasks.

A distilled 7B model is not a smaller copy of o3. It does not inherit every fact, tool skill, language ability, safety behavior, or long-context capability of the teacher.

It may be excellent at algebra and code repair while being worse at current events, legal nuance, or a five-step web task.

What it can inherit is a behavior: break the problem into parts, keep intermediate state, check a result, and only then answer.

That distinction matters because a quantized model with 7 billion parameters can be physically small enough for a laptop. Four-bit storage uses roughly half a byte per parameter.

The raw weights for a 7B model occupy about 3.5 GB before metadata and runtime memory. A 32B model needs roughly 16 GB just for raw four-bit weights.

The latter is not a comfortable fit on a typical 16 GB machine once the operating system, program, and attention cache join the party.

Quantization, which stores weights with fewer bits to reduce memory, helps with deployment. It does not teach a model to reason.

Distillation is the teaching step. You can see the broader mechanics in the lesson on model distillation.

What reasoning distillation actually transfers

Classic knowledge distillation trains a small student, the model you want to ship, to imitate a large teacher, the model that already performs well.

The student does not receive the teacher’s weights. It receives training signals produced by the teacher.

There are several forms of that signal. A teacher can provide logits—unnormalized scores—or soft targets—a probability distribution derived from those logits.

Both can be used as distillation signals. A teacher can also generate complete answers for a student to imitate.

Reasoning distillation usually means the second approach with an important addition: the generated answer contains intermediate reasoning, not just the final response.

A reasoning trace is a sequence of intermediate steps between a question and its answer. It might contain arithmetic, a plan, a code edit, a contradiction, or a self-check.

It is often called a chain of thought, although a teacher-generated trace is not necessarily a faithful transcript of hidden cognition.

One important boundary matters in production: an accessible teacher-generated rationale is not the same thing as private reasoning tokens.

This workflow requires a teacher whose outputs are available and whose use for distillation is permitted, such as an open model you run yourself or an API that provides a structured rationale.

A provider may return only a final answer or a reasoning summary; a summary is not the hidden trace. Raw o3 reasoning tokens cannot simply be assumed to be downloadable.

Provider terms may also restrict using outputs to train another model, so check them before building the dataset.

For the retry problem, a conceptual training record could look like this:

{
  "prompt": "An API receives 1200 requests. Three percent fail, and each failed request is retried exactly once. How many total attempts occur?",
  "trace": "Three percent of 1200 is 36 failed requests. The original 1200 attempts still count. Each of the 36 failures creates one retry, so the total is 1200 plus 36, or 1236.",
  "answer": "1236"
}

The student is trained with supervised fine-tuning, meaning its parameters are adjusted so that the teacher’s trace and answer become likely continuations of the prompt.

At each position, the training objective rewards the next target token given the prompt and the earlier target tokens.

That gives the student many learning points inside one example. With answer-only training, the target might be just 1236.

With a trace, the target includes “three percent,” “36 failures,” “one retry,” and “total.” The student is not merely rewarded for landing on the destination. It is shown a route.

Suppose 100,000 examples have two answer tokens each, but their traces average 80 target tokens. Answer-only training supplies about 200,000 target positions.

Trace training supplies about 8 million. That is 40 times as many positions for the optimizer to learn from.

It does not mean the trace is 40 times more truthful or that quality automatically improves by 40 times. It means the supervision is less sparse.

At inference time, the teacher is gone. The student generates its own intermediate tokens and then its answer.

Those tokens cost time. A 500-token reasoning trace takes more computation than a 20-token answer, and the model’s key-value cache, which stores attention information from earlier tokens, grows as the trace grows.

Distillation moves the expensive teacher work offline; it does not make thinking free.

Why a smaller student can beat a larger ordinary model

A large general-purpose model has capacity for many behaviors. That does not mean every behavior is equally easy for it to produce.

A model trained mainly to answer concisely may know the facts needed for the retry problem but skip the decomposition.

A distilled student is trained on a narrower target distribution. It sees thousands of examples where the desired behavior is explicit: identify the base quantity, calculate the rate, account for the retry, check the total.

The student may have fewer parameters but a stronger habit for this particular job.

That is why a smaller reasoning model can beat a much larger non-reasoning model on a math or coding benchmark. The comparison is not simply “small brain versus big brain.”

It is also “specialized problem-solving policy versus general answer production.”

The January 2025 DeepSeek-R1 release made this visible. DeepSeek published dense R1-Distill models ranging from 1.5B to 70B parameters, based on Qwen and Llama model families and trained using reasoning data generated by R1.

The family showed that small models could be surprisingly strong on selected mathematics and coding evaluations, sometimes outperforming much larger models that had not been trained to spend tokens on explicit reasoning.

The correct conclusion is narrower than the headline. A 1.5B distill is not equivalent to R1 or o3 across the board.

It can, however, carry a useful slice of the teacher’s problem-solving behavior into a model that is cheaper to run and easier to keep local. For a fixed workload, that can be the difference between an idea and a product.

The economics are straightforward. Imagine one million user requests that would each make a frontier model generate 4,000 output tokens.

That is 4 billion output tokens served online. Instead, you might generate a much smaller teaching set offline, train a student once, and serve short local requests thereafter.

The offline generation is still expensive. It is simply paid once and amortized across many inferences.

A trace is not a proof

This is the strongest objection to reasoning distillation: perhaps the student learns the style of reasoning rather than the underlying procedure.

That objection is not a technicality. It is often exactly what happens.

A teacher may produce a long trace because long traces correlate with correct answers, not because every sentence caused the answer.

If those traces contain arithmetic mistakes, invented facts, or unnecessary rhetorical filler, the student can learn all three. It may respond to the retry problem with a polished paragraph that sounds analytical while still copying 36 as the final total.

Pure imitation also has a ceiling. If the teacher never demonstrates how to handle a new kind of problem, the student has little basis for discovering that capability.

Distillation compresses demonstrated behavior. It does not transfer the teacher’s entire internal state.

Verify the trace, not just the answer

There are ways past the ceiling, but they are not magic. Generate several candidate traces, but separate final-answer filtering from process verification.

• A correct final answer does not certify the intermediate trace. Wrong steps can cancel out, a calculator may check only the final arithmetic, and a unit-test suite may miss incorrect or unsafe code.
• Check intermediate arithmetic or symbolic steps where possible.
• Execute code against hidden and adversarial tests in a sandbox.
• Use structured intermediate fields, such as storing failure_count, retry_count, and total_attempts separately for the retry example.
• Obtain human review for reasoning that cannot be verified mechanically.
• Discard or mask unverified traces rather than training on them as if they were correct.

Then train on the verified examples.

Reinforcement learning with a verifiable reward can push a student toward solutions that pass the checker rather than solutions that merely resemble the teacher’s prose.

A student can sometimes outperform the teacher on a narrow benchmark after this filtering.

That usually means the student learned from a cleaner distribution than a single noisy teacher sample, or became specialized to the benchmark. It is not evidence that a 7B model acquired all of the teacher’s general intelligence.

The most common practical failure is easy to spot. The student handles the exact training template, answering 1,236 for the original problem, but fails when the numbers change:

• 2,500 requests
• 2.4 percent failures
• one retry per failure

The correct calculation is 60 failures and 2,560 total attempts. A model that repeats 1,236, calculates 2.4 percent of 1,200, or produces a long answer with no stable arithmetic is memorizing the surface form.

The cure is varied wording, varied numbers, held-out cases, and an actual verifier. More elegant prompts will not repair contaminated training data.

What to do on Monday morning

Define the task and dataset

Start with the task envelope, not with a model. Write down what the local student is supposed to handle and what it must reject or defer.

“Reasoning” is too broad. “Calculate retry totals from structured incident reports and explain the arithmetic” is testable.

Create a locked evaluation set before generating teacher data. Two hundred real cases is a useful starting point if you have them: ordinary cases, edge cases, changed numbers, misleading wording, and a few cases where the right action is “insufficient information.”

Keep these examples out of training. Read the guidance on LLM evaluations before choosing a single score; exact correctness, refusal quality, latency, and memory use all matter.

Next, ask the teacher for several candidate solutions for each training prompt. Three to eight candidates is a practical range, not a law.

Select with the strongest checker available:

• Execute code.
• Recalculate arithmetic.
• Compare structured fields.
• Have a person review answers where correctness cannot be mechanically decided.

Do not filter only by the teacher’s confidence. Confidence is another model output, not a proof.

Keep the training target boring

Keep the training format boring. Include the prompt, a concise reasoning trace, the final answer, and any useful verification result.

Remove private data and irrelevant teacher chatter.

You do not need to show the trace to the end user. Supervised fine-tuning does not automatically hide it: if the trace is part of the target, the student may emit it, and a generated statement such as verified is self-report, not verification.

Serialize the trace and answer with explicit delimiters. Parse or suppress the trace at output time, and run an independent verifier after extracting the answer.

The final check must be executed by a trusted tool or service, not merely generated by the student.

A product can train on a detailed teaching trace and return a short answer with a final check.

Measure the deployment trade-off

Fine-tune a student that fits the target device, then measure the unquantized student before applying deployment compression.

Compare it with the original small model on the locked set. After that, test the quantized version.

The lesson on quantization covers why lower-bit weights save memory but can change output quality.

A model that fits in RAM but takes 18 seconds to answer a routine request has not really solved the product problem.

Compare fixed cost and tuned quality

Keep the semantic prompt and evaluation data constant, but use each model’s correct chat template.

Report both regimes: a controlled fixed-token or fixed-latency comparison, including truncation rates, and a per-model tuned comparison at matched quality, latency, and memory targets.

In the controlled regime, state the shared output budget or latency cap. In the tuned regime, allow model-specific decoding settings and token budgets rather than assuming identical settings are optimal.

This separates a fair fixed-cost comparison from a production-quality comparison.

Record:

• accuracy
• average and p95 latency
• peak memory
• output length
• fallback rate

P95 means the time below which 95 percent of requests finish; it catches the ugly tail that an average hides.

Also test changed numbers and changed wording. Those are where memorization first shows its face.

Finally, give the student an escape hatch. If the verifier rejects the result, the request requires current information, or the question falls outside the task envelope, route it to a larger model or a human.

A local reasoning model is often excellent as a first pass, a classifier, or a planner. It does not need to be the final authority on every question.

That is the real bargain. The frontier model pays the discovery cost.

The student remembers a useful procedure and performs it locally, repeatedly, and cheaply. For the retry calculation, that may be all you need. For every problem in the world, it is not even close.