Scaling laws & compute-optimal training
Two LLMs, the same compute budget — 280B parameters or 70B. The smaller one wins. How test loss falls as a predictable power law in model size, data, and compute, and why the Chinchilla rule of ~20 tokens per parameter reshaped how LLMs are trained.
Written and reviewed by Shreyash Prashu Last reviewed
What you'll learn
- Why test loss falls as a smooth power law in parameters, data, and compute
- The C ≈ 6ND compute estimate and the ~20-tokens-per-parameter rule
- Compute-optimal training (Chinchilla) vs the older "just make it bigger"
- Why modern models deliberately over-train past the optimum
Before you start
Here is a result that surprised the whole field. Take two language models trained on the same compute budget. One has 280 billion parameters; the other has 70 billion — four times smaller. The smaller model wins, clearly and across the board. Understanding why is the difference between burning a compute budget and spending it.
The discovery: loss is a power law
When researchers trained models across many orders of magnitude, they found something almost too clean: test loss falls as a smooth power law in three quantities:
- the number of parameters
N - the amount of training data
D - the total compute
C
Plot loss against compute on log–log axes and you get a near-straight line that holds across a millionfold range.
The practical magic of a power law is prediction: measure a few small models, fit the curve, and you can forecast the loss of a model 100× bigger before spending the money to train it. Scaling stopped being a gamble and became engineering.
The compute budget: C ≈ 6ND
Training compute is well approximated by a simple formula — about 6 FLOPs per parameter per token:
C ≈ 6 · N · Dwhere N is parameters and D is training tokens. This is the lever that frames
the central question: for a fixed C, you can spend it on a bigger model
(raise N) or more data (raise D) — but not both. So which?
Compute-optimal: the Chinchilla correction
The first scaling-laws work (2020) leaned toward “make the model bigger,” and the field built giants trained on relatively little data. In 2022 the Chinchilla study redid the experiment carefully and found those giants were badly under-trained — starved of data for their size. The fix is a now-famous rule of thumb: train on roughly 20 tokens per parameter.
The headline comparison, at essentially the same compute:
| Model | Params N | Tokens D | Compute 6ND | Tokens / param |
|---|---|---|---|---|
| Gopher | 280 B | 300 B | ≈ 5.0 × 10²³ | 1.1 |
| Chinchilla | 70 B | 1.4 T | ≈ 5.9 × 10²³ | 20 |
Same budget. Chinchilla is 4× smaller but trained on 4.7× more data — and it beats Gopher decisively.
Gopher spent its budget on parameters it never had enough data to teach. The
lesson: a bigger model is not better if you can’t feed it. For a fixed compute
budget the optimum scales as N* ∝ √C and D* ∝ √C — so doubling compute means
making the model about 1.4× bigger and feeding it 1.4× more data, together.
Pause and think
GPT-3 was 175 B parameters trained on 300 B tokens — about 1.7 tokens per parameter. By the Chinchilla rule, was it over- or under-trained, and which knob would you turn to fix it on the same budget?
It was severely under-trained: at 175 B parameters, compute-optimal wants ~3.5 trillion tokens, not 300 billion. The fix is far more data (or a smaller model), not more parameters.
Emergence — what scale buys
Beyond a smoothly falling loss, scale also unlocks capabilities. Some abilities appear only once a model passes a size threshold, then improve sharply:
- multi-step arithmetic
- in-context learning
- following novel instructions
You can’t always predict which ability appears when. But the reliable downward march of loss is why the field keeps scaling, and why “just make it bigger, and feed it enough” has been the dominant strategy of the LLM era.
In one breath
- Test loss falls as a smooth power law in parameters N, data D, and compute C — straight on log–log axes across many orders of magnitude, so you can predict a big model’s loss from small ones.
- Training compute is about C ≈ 6ND, which forces a choice: for a fixed budget, spend on a bigger model or more data.
- Chinchilla showed the early giants were under-trained; compute-optimal is roughly 20 tokens per parameter.
- At equal compute, 70B-on-1.4T (Chinchilla) beats 280B-on-300B (Gopher) — a bigger model you can’t feed is wasted; the optimum scales as N ∝ √C and D ∝ √C together.
- Modern models over-train past the optimum (Llama 3 8B ≈ 1,900 tok/param) to get smaller models that are cheaper to serve.
Check yourself
Quick check
What to remember
- LLM test loss is a power law in parameters, data, and compute — predictable enough to forecast big models from small ones.
- Training compute is about C ≈ 6ND, so a fixed budget trades model size against data.
- Chinchilla: the compute-optimal balance is ~20 tokens per parameter; the early giants (Gopher, GPT-3) were under-trained.
- Production pushes past the optimum to cut serving cost, and scale also unlocks emergent capabilities — why bigger-and-better-fed has defined the LLM era.
Practice this in an interview
All questionsSmaller models win on latency, inference cost, on-device deployment, and fine-tuning feasibility. When trained on high-quality, curated data and aligned for a narrow task, a 7B–13B model can match or exceed a general-purpose 70B+ model on that specific workload while using a fraction of the compute budget.
Work top-down: start at the model layer with quantization, distillation, or routing cheaper models for easy requests, since model choices drive every downstream cost. Then optimize the runtime with batching, caching, and techniques like prompt caching for LLMs, and finally match infrastructure to the load using autoscaling on queue depth and spot or batch capacity. Track cost per token or per prediction alongside latency percentiles and accuracy so optimizations never silently degrade quality.
Cost scales with input plus output tokens; latency scales with output tokens and model size. The highest-leverage levers are: model routing (use a small model when the task is simple), prompt caching (reuse expensive prefix computation), output length control, and batching. Together these can cut spend 60–90% without quality regression.
Parameters are the learnable floating-point numbers — weights and biases — that define a neural network's behaviour. In a transformer LLM, they are distributed across token embedding matrices, multi-head attention projection matrices (Q, K, V, O), and feed-forward network layers. They encode compressed statistical associations between tokens learned during training, not explicit facts or rules.