Pretraining — where an LLM's knowledge comes from

When ChatGPT tells you the capital of France, it did not learn that from your prompt. It learned it — along with grammar, arithmetic, Python, and a rough model of how the world works — from trillions of tokens of text, in a phase that finished long before you ever typed a word. That phase is pretraining, and it is where almost all of an LLM’s knowledge and compute live.

The puzzle: knowledge from a guessing game

Here’s what should feel strange. The entire pretraining objective is absurdly simple: predict the next token. Given “The cat sat on the,” guess what comes next. That’s it. No labelled facts, no curated question–answer pairs.

So how does a next-word guessing game produce something that can explain photosynthesis or debug your code? Because to predict the next word well, the model is forced to learn the structure underneath — grammar, facts, and reasoning are all just patterns that make the next token more predictable.

Self-supervised: the labels are free

The reason this scales to trillions of tokens is that it needs no human labels. Take any sentence from the internet, hide the next word, and you have a training example whose correct answer is simply the word you hid. The text labels itself. This is self-supervised learning, and it’s why pretraining can consume the open web, code, and books without an army of annotators.

The model reads a window of tokens and outputs a probability distribution over the whole vocabulary for the next token:

Cross-entropy and perplexity — the score

The training loss is cross-entropy: the negative log of the probability the model assigned to the actual next token. Suppose the true next word was “mat” and the model gave it probability 0.40:

loss = -ln(0.40) = 0.92 nats

If instead it had been only 0.10 sure, the loss is -ln(0.10) = 2.30 — much higher. The model is punished in proportion to how surprised it was by the truth, so training pushes it to put probability where the real next tokens land.

A friendlier way to read the same number is perplexity — exp(loss):

perplexity = exp(0.92) ≈ 2.5

Perplexity is “how many tokens the model was effectively choosing between.” A perplexity of 2.5 means it was about as unsure as a fair pick among 2.5 options; the 0.10 case gives exp(2.30) = 10, far more lost. Lower perplexity = sharper, more correct guesses, and watching it fall is how you watch a model learn.

The scale — why it’s the expensive part

Pretraining is enormous. Modern models train on trillions of tokens — Llama 3 saw roughly 15 trillion — scraped from the web, code, and books, then heavily deduplicated and quality-filtered (garbage in genuinely means garbage out). This single phase eats the overwhelming majority of the total compute that goes into a model; everything after it is comparatively cheap.

That’s why pretraining is run once, at great expense, by a handful of labs — and why everyone else adapts the result rather than repeating it.

Where pretraining sits in the pipeline

A raw pretrained model is a brilliant autocomplete — it continues text, but it won’t reliably follow an instruction or refuse a harmful one. Two cheaper stages turn it into an assistant:

1. Pretraining — next-token prediction on trillions of tokens. Builds knowledge and fluency. (The expensive ~99% of compute.)
2. Supervised fine-tuning (SFT) — train on curated instruction → response examples so it learns to follow instructions.
3. Alignment (RLHF / DPO) — tune on human preference data so its answers are helpful and harmless. (See Alignment: SFT, RLHF & DPO.)

In one breath

• Pretraining teaches an LLM by next-token prediction over trillions of tokens — to guess the next word well, it must learn grammar, facts, and reasoning.
• It’s self-supervised: every token in the corpus is a free training example, so it needs no human labels and can consume the whole web.
• The loss is cross-entropy (−ln of the probability given to the true token); perplexity = exp(loss) reads as “how many tokens it was choosing between” — lower is better.
• It’s by far the most expensive phase (Llama 3: ~15T tokens), run once by a few labs on heavily filtered data.
• A pretrained base model is an autocomplete; SFT then alignment turn it into an instruction-following, helpful assistant.

Check yourself

What to remember

• Pretraining is next-token prediction over trillions of self-supervised tokens — the source of an LLM’s knowledge and the bulk of its compute.
• The objective is cross-entropy loss; perplexity = exp(loss) is the readable version — lower means sharper next-token guesses.
• Data is web + code + books, deduplicated and quality-filtered at massive scale.
• The output is a base model; SFT and alignment turn it into a helpful, instruction-following assistant — and scaling laws tell you how big to make it.