Language models before transformers

A language model (LM) does one thing: predict the next word given the previous ones. It’s the oldest task in NLP — and, remarkably, the exact objective a modern LLM like GPT trains on. What changed over the decades isn’t the goal; it’s the machinery. Before transformers, language modeling meant counting n-grams and, later, recurrent networks.

The n-gram model: count and generate

A true language model would condition on all prior words — impossible to count. The n-gram model makes the Markov assumption: the next word depends only on the last n−1 words. With that, you can just count: estimate P(next | previous words) from how often each continuation appeared in a corpus. A bigram model (n=2) conditions on one previous word:

from collections import defaultdict, Counter

corpus = "the cat sat the cat ran the dog sat".split()
bigrams = defaultdict(Counter)
for a, b in zip(corpus, corpus[1:]):
    bigrams[a][b] += 1                       # count each (prev -> next)

def generate(start, n=5):                    # walk the chain: pick the most likely next word
    out = [start]; w = start
    for _ in range(n):
        if w not in bigrams: break
        w = bigrams[w].most_common(1)[0][0]
        out.append(w)
    return out

print("P(next | 'the'):", dict(bigrams["the"]))
print("generated:", " ".join(generate("the")))

P(next | 'the'): {'cat': 2, 'dog': 1}
generated: the cat sat the cat sat

After the, the corpus saw cat twice and dog once, so P(cat | the) = 2/3. Generation just walks this Markov chain — and notice the output immediately loops (the cat sat the cat sat). That repetition is the visible symptom of the model’s core weakness.

Two problems: sparsity and short context

n-gram models fail in two ways:

• Sparsity. Most possible n-grams never appear in any corpus, so their count — and probability — is zero, and the model assigns probability 0 to any sentence containing one. The fix is smoothing: shave probability mass off seen n-grams and give it to unseen ones (add-k, backoff to shorter n-grams, Kneser-Ney). And the vocabulary explosion is brutal: a trigram model over a 50k vocabulary has 50,000³ possible entries.
• Short context. A bigram sees one word back; even a 5-gram sees four. Anything longer-range — agreement across a clause, a topic set up sentences ago — is invisible, which is exactly why the generated text loops and drifts. You can’t fix this by raising n: sparsity explodes faster than context grows.

Language models are scored by perplexity — roughly, how surprised the model is by real text (2 raised to the per-word cross-entropy). Lower is better; a perplexity of 100 means the model is as unsure as if choosing uniformly among 100 words at each step.

In one breath

• A language model predicts the next word — the oldest NLP task and the exact objective GPT trains on; only the machinery changed.
• The n-gram model makes the Markov assumption (next word depends on the last n−1), so you count P(next | prev) from a corpus and generate by walking the chain (the demo loops the cat sat…).
• Sparsity: unseen n-grams get probability 0, fixed by smoothing (add-k, backoff, Kneser-Ney); the n-gram space also explodes (50k³ trigrams).
• Short context: an n-gram sees only n−1 words back, so long-range structure is invisible — and you can’t just raise n (sparsity explodes). LMs are scored by perplexity (lower = less surprised).
• RNN LMs learn dense representations with unbounded context (better), and transformers fixed long-range — but n-gram → RNN → GPT is one objective, improved architectures.

Quick check

Next

The objective here, scaled up with a better architecture, is GPT pretraining. The leap past counting is the RNN/LSTM language model, and long-range modeling is the transformer. The chain-walking rests on Markov chains.