Model compression: pruning and distillation
How to trade redundant weights, numerical precision, or teacher guidance for a model that actually fits and runs in production.
What you'll learn
- Why unstructured sparsity often changes a model file without changing wall-clock latency
- How magnitude pruning and iterative prune-then-finetune work
- Why soft targets and temperature make knowledge distillation useful
- How pruning, distillation, and quantization trade different resources
- How to measure a real compression frontier using deployment metrics
Before you start
Your image classifier works beautifully in the lab. Then you put it on the device that must run it.
The model has 25 million parameters stored as 32-bit floats. That is about 100 MB before runtime overhead. Its p95 latency, meaning the time taken by 95 percent of requests, is 180 ms on the device. Your product budget is 50 MB and 80 ms. The device has four CPU cores, limited memory bandwidth, and no patience.
Deleting random weights may make a report say “50 percent sparse.” It may not make one prediction arrive any sooner.
Model compression makes a model smaller, faster, or cheaper by removing information the model can survive without:
- Pruning removes parameters or network structures.
- Distillation trains a smaller student to imitate a larger teacher.
- Quantization represents weights or activations with fewer bits. It reduces compute cost only when the deployment runtime has supported low-precision kernels.
They reduce different costs and fail in different ways.
Three things you can trade away
Pruning trades away parameters. A parameter is a learned number, such as a connection weight. If it is close to zero, setting it to zero may barely change the output. The hardware must still know how to skip it: a zero inside an ordinary dense matrix is often loaded and multiplied like any other value.
Distillation trades away capacity. A smaller student has fewer layers, channels, or hidden units, so it cannot represent everything the teacher can. Training guides it toward the teacher’s useful behavior.
Quantization trades away numerical precision. A 32-bit float represents more values than an 8-bit integer. Fewer bits reduce storage and memory traffic, but rounding and limited range can damage accuracy.
The useful question is not “Which method compresses the most?” It is “Which resource is breaking my deployment?” Serialized weight size is only part of memory use; activations, runtime buffers, metadata, and preprocessing also matter.
Pruning: zeros are not automatically speed
Pruning sets selected parameters to zero. The simplest rule is magnitude pruning: rank weights by absolute value and remove the smallest ones. A weight of -0.0003 is a more obvious candidate than one of 1.7, because changing the former usually changes the layer output less.
That rule is only local. Small weights can cooperate, and a seemingly important weight can become unnecessary after its neighbors change. Fine-tuning lets the remaining weights compensate.
There are two importantly different kinds of sparsity.
Unstructured pruning
Unstructured pruning removes individual weights:
[ 0.8 0.0 -0.2 0.0 ]
[ 0.0 1.1 0.0 0.4 ]The matrix keeps the same rows and columns. This preserves the architecture and can produce impressive sparsity percentages.
But standard dense matrix multiplication does not usually inspect every value and branch around zeros. It runs the same dense kernel over the same rectangular array. After pruning, you may have a matrix containing zeros rather than a smaller matrix.
Unstructured pruning can reduce storage, support compressed formats, or work with hardware designed for a particular sparsity pattern. Do not turn a sparsity percentage into a latency claim.
Structured pruning
Structured pruning removes objects hardware already understands:
- whole convolution channels or filters;
- rows or columns of a linear layer;
- attention heads or blocks of weights.
The tensor itself becomes smaller. A convolution with 64 output channels can become one with 40, and the next layer must be changed consistently. Because the shape changes, ordinary dense kernels perform fewer operations on a smaller tensor.
The trade-off is less flexibility. A channel may contain useful features for many examples, even if its average importance is modest. Removing it can hurt more than removing the same number of scattered weights. Deleting an attention head helps only if the implementation removes its dimensions rather than masking the head.
The prune-and-recover loop
A practical pruning run is:
- Train a dense baseline to a known quality level.
- Score weights or structures by an importance rule.
- Prune a portion.
- Fine-tune until validation quality recovers.
- Repeat until the deployment or quality limit is reached.
- Export the genuinely smaller or sparse representation.
This is iterative prune-then-finetune. It often beats one-shot pruning at the same final sparsity because the model gets several chances to reorganize. “Prune 50 percent” could mean removing half the weights once or removing smaller portions while repeatedly adapting; those are different experiments.
A pruning mask is not a compact deployment format. In PyTorch, removing the pruning reparameterization makes zeros permanent but leaves the original rectangular tensor. The zero fraction and measured latency therefore remain separate facts.
Distillation: teach behavior, not every parameter
Suppose a teacher sees a package image and produces three logits, or raw class scores:
crushed corner: 4
torn label: 2
no damage: 0Ordinary softmax gives approximately:
crushed corner: 0.867
torn label: 0.117
no damage: 0.016The hard label says only “crushed corner.” It discards the fact that the teacher considers “torn label” much more plausible than “no damage.” That ranking can encode visual similarity, class boundaries, and useful uncertainty.
Knowledge distillation trains a smaller student to match the teacher’s outputs. A soft target is a probability distribution over classes rather than a one-hot answer.
A temperature controls how flat the distribution becomes. At temperature 4, divide the logits by 4 before softmax:
crushed corner: 1
torn label: 0.5
no damage: 0The probabilities become approximately:
crushed corner: 0.506
torn label: 0.307
no damage: 0.186The softer distribution exposes relationships hidden by hard labels. A common objective combines hard-label cross-entropy with teacher-student Kullback–Leibler divergence, scaling the distillation term by T²:
loss = (1 - alpha) * CE(hard_label, student_logits) + alpha * T^2 * KL(teacher_probs_T, student_probs_T)
alpha controls how much the student listens to the teacher, and T is the temperature. Neither has a universally correct value. Retain the hard-label term when ground truth is available because the teacher can be confidently wrong.
Why can an 8-million-parameter student beat another 8-million-parameter model trained from scratch? The teacher supplies a richer target on every example: not only which class won, but how it orders the alternatives. That can guide optimization and act as a learned regularizer. The advantage is empirical, not guaranteed; a poor teacher, undersized student, or unsuitable temperature can make distillation worse.
Unlike pruning, distillation lets the student change its internal architecture while preserving the input-output interface. For LLM-specific examples and token-level distillation, see distillation.
Quantization: fewer bits, fewer bytes moved
Quantization represents numbers with lower precision. Moving a weight tensor from float32 to int8 changes each weight from 4 bytes to 1. Scales and sometimes zero points map the integer back to an approximate real value.
The main mechanism is often memory traffic: smaller weights move through memory and caches more cheaply. Many CPUs, GPUs, and accelerators also provide specialized int8 or float16 instructions.
- Post-training quantization is cheap but can hurt sensitive layers.
- Quantization-aware training simulates rounding during training so the model learns to tolerate deployment errors.
Quantizing weights alone reduces model memory while leaving activation memory or compute mostly unchanged. Quantizing weights and activations can improve speed further, but activation ranges vary with inputs and make the process harder.
This differs from mixed-precision training, which uses lower precision during training while retaining selected higher-precision values for stability. See mixed precision. For LLM-focused methods, see quantization.
The compression frontier is a measurement problem
There is no single best compressed model. There is a compression-accuracy frontier: models for which improving one important objective requires worsening another.
Measure the product metric, not just average accuracy:
- quality on important slices and costly classes;
- calibration when probabilities drive decisions;
- p50 and p99 latency;
- peak RAM or VRAM, not only serialized weight size;
- throughput at the real batch size;
- energy per request when power matters;
- end-to-end time, including transfers and preprocessing.
Compare models on the same device, operating system, compiler, thread count, batch size, input shape, and warm-up procedure. A desktop GPU benchmark does not answer an Android CPU question, and batch size 32 does not answer a one-request-at-a-time question.
Memory bandwidth can dominate batch-one inference. A 100 MB float32 model must read roughly 100 MB of weights for one inference. At 50 GB/s sustained bandwidth, that weight read alone has a floor near 2 ms. An int8 version with 25 MB of weights has a corresponding floor near 0.5 ms. Real latency is higher because of cache misses, activations, synchronization, and computation. An unstructured-pruned model that remains a 100 MB dense tensor can still incur roughly the same traffic; a physically smaller structured or quantized model may reduce it.
Choosing among the options
Start with the constraint that is failing.
| Hard constraint | First option to test | Why it fits | Realistic alternative |
|---|---|---|---|
| Weight memory and fast low-precision kernels | Quantization | Fewer bits reduce bytes moved and stored | Distill to a smaller architecture |
| Wall-clock latency and changeable tensor shapes | Structured pruning | Dense kernels compute on smaller tensors | Distillation or a smaller model |
| Same architecture and an available sparse runtime | Unstructured pruning | Individual weights can be removed without changing interfaces | Quantization |
| Smaller model with the same input-output behavior | Distillation | A student learns the teacher’s useful behavior | Transfer learning |
| Predictable engineering | Smaller architecture | It avoids compression artifacts and special kernels | Train a teacher, then distill |
Combinations often work: distill the teacher into a smaller student, then quantize it; or prune channels, fine-tune, and quantize the resulting dense model. Validate after every stage rather than assuming the effects add neatly.
Failure modes
Sparsity improves but latency does not. The dense kernel is probably unchanged. Export a physically smaller structured model or use a supported sparse kernel, then benchmark again.
One-shot pruning causes a sharp accuracy drop. Use smaller iterative steps, fine-tune after each, and inspect layer or slice sensitivity.
The student matches average accuracy but misses expensive cases. Mix hard-label and distillation losses, tune T and alpha, and select checkpoints using the business metric rather than teacher agreement alone.
Quantization makes production slower. Check for float fallbacks, device copies, or dequantization at every layer. Measure the complete request path with a supported backend.
A small file still causes out-of-memory errors. Profile peak activations, temporary workspaces, and batch size. Weight compression does not automatically compress activations.
What to remember
- Pruning, distillation, and quantization discard parameters, capacity, and precision respectively.
- Unstructured zeros often do not speed up ordinary dense kernels.
- Structured pruning changes tensor shapes, enabling real dense-kernel speedups.
- Soft targets preserve relationships hidden by hard labels, but distillation is not guaranteed to help.
- Choose the model on the quality-latency-memory frontier measured on the relevant hardware.
Quick check
Practice this in an interview
All questionsModel quantization represents weights and sometimes activations with fewer bits, reducing memory use and often improving inference cost or latency. More aggressive formats such as INT4 can reduce accuracy, but calibration, per-group scaling, outlier handling, and selective higher-precision layers can preserve quality; the result must be measured on the target workload and hardware.
Work top-down: start at the model layer with quantization, distillation, or routing cheaper models for easy requests, since model choices drive every downstream cost. Then optimize the runtime with batching, caching, and techniques like prompt caching for LLMs, and finally match infrastructure to the load using autoscaling on queue depth and spot or batch capacity. Track cost per token or per prediction alongside latency percentiles and accuracy so optimizations never silently degrade quality.
I would group deep-network regularization into weight penalties, stochastic methods, data and target transformations, and training controls. L2 weight decay, L1 penalties, dropout, augmentation, label smoothing, mixup, and early stopping reduce memorization by constraining solutions, injecting useful variation, or stopping before the model fits noise.
A flat or erratic loss almost always indicates a bug — in data loading, label encoding, loss function, or gradient flow — not an insufficiently tuned learning rate. Systematic debugging means isolating each component and verifying it works on a tiny, controlled example before scaling up.