Multi-token prediction

The entire training objective of a standard LLM is to predict one token: given everything so far, what comes next? It works spectacularly — but it has two costs. In training it is myopic: the model optimises one step at a time and is never directly rewarded for thinking a few tokens ahead, which is exactly what coding and reasoning need. In inference it is wasteful: every expensive forward pass through the whole network produces a single token, then you do it all again.

Multi-token prediction (MTP) attacks both. Instead of one next-token head, the model has k heads that predict the next k tokens from the same hidden state. That one change buys a denser training signal and a faster decoder.

k heads on a shared trunk

The architecture is barely changed. The transformer trunk runs as usual and produces a hidden state hₜ at each position. Standard models feed hₜ to a single output head predicting token t+1. MTP attaches k heads to hₜ: head 1 predicts t+1 (the normal one), head 2 predicts t+2, and so on. Training minimises the sum of all k next-token cross-entropy losses.

Payoff 1 — a denser, less myopic training signal

Single-token training gives the trunk one bit of feedback per position: was t+1 right? MTP gives it k. To predict t+2 and t+3 correctly, the hidden state must encode information about where the sequence is going, not just the immediate next word — so the trunk learns representations that look ahead. The result is better sample efficiency and notably stronger performance on tasks where planning matters, code and math especially. (The extra heads are an auxiliary objective; at the end of training you can keep just the main head and lose nothing on quality.)

Payoff 2 — a built-in draft model (faster inference)

The extra heads double as a draft model for speculative decoding, for free. In one forward pass the heads propose the next few tokens; the model then verifies those drafts in a single parallel pass and accepts the longest correct prefix. Because the draft heads share the trunk, you get speculation without training or running a separate small model — this is self-speculative decoding.

How much faster? If each drafted token is accepted independently with probability α and you draft K tokens, the expected number of tokens produced per verification step is (1 − α^(K+1)) / (1 − α):

def expected_tokens(accept, K):
    return (1 - accept ** (K + 1)) / (1 - accept)   # speculative-decoding expectation

for accept, K in [(0.8, 3), (0.7, 3), (0.9, 4)]:
    e = expected_tokens(accept, K)
    print(f"accept={accept}, draft heads K={K}  ->  {e:.3f} tokens / pass  (~{e:.2f}x)")

accept=0.8, draft heads K=3  ->  2.952 tokens / pass  (~2.95x)
accept=0.7, draft heads K=3  ->  2.533 tokens / pass  (~2.53x)
accept=0.9, draft heads K=4  ->  4.095 tokens / pass  (~4.10x)

With a realistic 80% acceptance and three draft heads you generate nearly 3 tokens per forward pass instead of one — the same output, the same model, roughly 3× the decoding throughput. Acceptance rates climb on predictable text (code, boilerplate), so the speedup is largest exactly where generation is most repetitive.

In one breath

• Standard LMs predict one token at a time: myopic in training (no direct look-ahead) and wasteful at inference (one token per full forward pass).
• Multi-token prediction attaches k heads to the shared trunk’s hidden state, predicting t+1 … t+k; training minimises the sum of the k next-token losses.
• Payoff 1 — denser signal: predicting several tokens forces look-ahead representations, improving sample efficiency and especially code/math; the extra heads are auxiliary (keep only the main head at deployment if you wish).
• Payoff 2 — self-speculative decoding: the extra heads are a built-in draft model — draft K, verify in one pass, accept the longest correct prefix — giving (1−α^(K+1))/(1−α) tokens per pass (~2.95× at α=0.8, K=3) with no separate model.
• Used by Meta’s MTP (parallel heads) and DeepSeek-V3 (sequential causal MTP).

Quick check

Next

Multi-token prediction is a frontier training-and-inference technique; the general inference method it feeds is speculative decoding, and the training economics it improves are governed by scaling laws.