Scaling laws & compute-optimal training
Two LLMs, the same compute budget — 280B parameters or 70B. The smaller one wins. How test loss falls as a predictable power law in model size, data, and compute, and why the Chinchilla rule of ~20 tokens per parameter reshaped how LLMs are trained.
What you'll learn
- Why test loss falls as a smooth power law in parameters, data, and compute
- The C ≈ 6ND compute estimate and the ~20-tokens-per-parameter rule
- Compute-optimal training (Chinchilla) vs the older "just make it bigger"
- Why modern models deliberately over-train past the optimum
Before you start
Here is a result that surprised the whole field. Take two language models trained on the same compute budget. One has 280 billion parameters; the other has 70 billion — four times smaller. The smaller model wins, clearly and across the board. Understanding why is the difference between burning a compute budget and spending it.
The discovery: loss is a power law
When researchers trained models across many orders of magnitude, they found
something almost too clean: test loss falls as a smooth power law in three
quantities — the number of parameters N, the amount of training data D, and
the total compute C. Plot loss against compute on log–log axes and you get a
near-straight line that holds across a millionfold range.
The practical magic of a power law is prediction: measure a few small models, fit the curve, and you can forecast the loss of a model 100× bigger before spending the money to train it. Scaling stopped being a gamble and became engineering.
The compute budget: C ≈ 6ND
Training compute is well approximated by a simple formula — about 6 FLOPs per parameter per token:
C ≈ 6 · N · Dwhere N is parameters and D is training tokens. This is the lever that frames
the central question: for a fixed C, you can spend it on a bigger model
(raise N) or more data (raise D) — but not both. So which?
Compute-optimal: the Chinchilla correction
The first scaling-laws work (2020) leaned toward “make the model bigger,” and the field built giants trained on relatively little data. In 2022 the Chinchilla study redid the experiment carefully and found those giants were badly under-trained — starved of data for their size. The fix is a now-famous rule of thumb: train on roughly 20 tokens per parameter.
The headline comparison, at essentially the same compute:
| Model | Params N | Tokens D | Compute 6ND | Tokens / param |
|---|---|---|---|---|
| Gopher | 280 B | 300 B | ≈ 5.0 × 10²³ | 1.1 |
| Chinchilla | 70 B | 1.4 T | ≈ 5.9 × 10²³ | 20 |
Same budget. Chinchilla is 4× smaller but trained on 4.7× more data — and
it beats Gopher decisively. Gopher spent its budget on parameters it never had
enough data to teach. The lesson: a bigger model is not better if you can’t feed
it. For a fixed compute budget the optimum scales as N* ∝ √C and D* ∝ √C —
so doubling compute means making the model about 1.4× bigger and feeding it
1.4× more data, together.
Pause and think
GPT-3 was 175 B parameters trained on 300 B tokens — about 1.7 tokens per parameter. By the Chinchilla rule, was it over- or under-trained, and which knob would you turn to fix it on the same budget?
It was severely under-trained: at 175 B parameters, compute-optimal wants ~3.5 trillion tokens, not 300 billion. The fix is far more data (or a smaller model), not more parameters.
Emergence — what scale buys
Beyond a smoothly falling loss, scale also unlocks capabilities: some abilities (multi-step arithmetic, in-context learning, following novel instructions) appear only once a model passes a size threshold, then improve sharply. You can’t always predict which ability appears when — but the reliable downward march of loss is why the field keeps scaling, and why “just make it bigger, and feed it enough” has been the dominant strategy of the LLM era.
In one breath
- Test loss falls as a smooth power law in parameters N, data D, and compute C — straight on log–log axes across many orders of magnitude, so you can predict a big model’s loss from small ones.
- Training compute is about C ≈ 6ND, which forces a choice: for a fixed budget, spend on a bigger model or more data.
- Chinchilla showed the early giants were under-trained; compute-optimal is roughly 20 tokens per parameter.
- At equal compute, 70B-on-1.4T (Chinchilla) beats 280B-on-300B (Gopher) — a bigger model you can’t feed is wasted; the optimum scales as N ∝ √C and D ∝ √C together.
- Modern models over-train past the optimum (Llama 3 8B ≈ 1,900 tok/param) to get smaller models that are cheaper to serve.
Check yourself
Quick check
What to remember
- LLM test loss is a power law in parameters, data, and compute — predictable enough to forecast big models from small ones.
- Training compute is about C ≈ 6ND, so a fixed budget trades model size against data.
- Chinchilla: the compute-optimal balance is ~20 tokens per parameter; the early giants (Gopher, GPT-3) were under-trained.
- Production pushes past the optimum to cut serving cost, and scale also unlocks emergent capabilities — why bigger-and-better-fed has defined the LLM era.