Distillation: Teaching a Small Model to Mimic a Big One

The wall quantization can’t break

Quantization made your 70B model fit in 40 GB. But it is still a 70B model — seventy billion multiply-adds for every single token you generate. On a phone, inside a browser, or under a tight latency budget, “smaller weights” is not enough. You need fewer weights.

That is a different problem with a different answer: train a brand-new, smaller model — fewer layers, fewer parameters — to copy what the big one does. The big model is the teacher; the small one is the student. This is knowledge distillation. The clever part is not the shrinking — it is how the student learns.

Learn from the whole answer, not just the pick

Here is the insight that makes it work. Show a well-trained classifier a photo of a dog. It does not just say “dog” — it outputs a probability for every class:

dog 90% · wolf 8% · cat 1.9% · ship 0.09% · car 0.01%

The hard label — the ground truth — is only “dog.” Train the student on that one-hot answer and you throw almost everything away. But look at what the teacher actually said: a dog is a little like a wolf, barely like a cat, and nothing like a car. Those ratios are real knowledge about how the world is shaped — Geoffrey Hinton called it “dark knowledge.” It is free supervision the student could never get from the bare word “dog.”

So distillation trains the student to match the teacher’s entire probability distribution — its soft labels — not just the winning class.

The temperature knob

There is a catch: a confident teacher’s distribution is too sharp to be useful. Ninety-nine percent on the right answer drowns out the interesting ratios underneath. So we soften it with a temperature T inside the softmax:

p_i = softmax(z_i / T)

where z are the teacher’s raw logits. At T = 1 you get the normal, spiky distribution. Raise T and the probabilities flatten, lifting the tiny ones into view — exactly the “dog is more like a wolf than a car” signal the student needs to learn. Drag the slider above and watch the dark knowledge appear. Teacher and student use the same T during training; at inference the student drops back to T = 1.

The distillation loss

The student is trained against two targets at once:

• Soft loss — match the teacher’s softened distribution. This is the KL divergence between the student’s and teacher’s T-softened outputs, scaled by T² (the scaling keeps the gradient magnitude steady as you change T).
• Hard loss — the ordinary cross-entropy against the true label, so the student stays anchored to ground truth even where the teacher is wrong.

L = α · L_soft(T)  +  (1 − α) · L_hard

α balances the two; a heavy weight on the soft term (around 0.9) is common, because the soft targets carry the richer signal.

Three flavours of distillation

Response-based is the classic above: the student matches the teacher’s final output distribution (its soft labels). Simple, model-agnostic, and most of what people mean by “distillation.”

Feature-based goes deeper: the student is also nudged to match the teacher’s intermediate activations — the hidden representations a few layers in — not just the final answer. This transfers more, but you must map the (differently shaped) layers between teacher and student. DistilBERT uses a version of this.

Sequence / data distillation is how most small language models are made today. You do not have the teacher’s logits for a closed model like GPT-4 — so instead you have it generate thousands of high-quality answers, and fine-tune the student on that synthetic dataset. The teacher’s “soft” knowledge is baked into the text it produces. Stanford’s Alpaca and the Orca line were built this way; so are countless small open models.

It works — the receipts

The headline number to remember: a good distillation keeps roughly 95–97% of the teacher’s quality at a fraction of the size — if the student is large enough and the data is good. Push the student too small and quality falls off a cliff; there is no free lunch, only a very good trade.

Quantization vs distillation

Both shrink a model, but they change completely different things — and that is exactly why you often want both.

The mental model: quantization makes each weight cheaper to store; distillation makes there be fewer weights to compute. Quantization saves memory and bandwidth; distillation saves actual FLOPs, which is the only thing that makes a model fundamentally faster.

And they compose. The strongest small models are usually distilled first, then quantized:

Quiz

Next

Speculative Decoding — keep your big model’s quality but get a small model’s speed, by letting a tiny draft model run ahead and having the big one check its work.