Language models before transformers

A language model (LM) does one thing: predict the next word given the previous ones. It’s the oldest task in NLP — and, remarkably, the exact objective a modern LLM like GPT trains on.

What changed over the decades isn’t the goal; it’s the machinery. Before transformers, language modeling meant counting n-grams and, later, recurrent networks.

The n-gram model: count and generate

A true language model would condition on all prior words — impossible to count. The n-gram model makes the Markov assumption: the next word depends only on the last n−1 words.

With that, you can just count: estimate P(next | previous words) from how often each continuation appeared in a corpus. A bigram model (n=2) conditions on one previous word:

from collections import defaultdict, Counter

corpus = "the cat sat the cat ran the dog sat".split()
bigrams = defaultdict(Counter)
for a, b in zip(corpus, corpus[1:]):
    bigrams[a][b] += 1                       # count each (prev -> next)

def generate(start, n=5):                    # walk the chain: pick the most likely next word
    out = [start]; w = start
    for _ in range(n):
        if w not in bigrams: break
        w = bigrams[w].most_common(1)[0][0]
        out.append(w)
    return out

print("P(next | 'the'):", dict(bigrams["the"]))
print("generated:", " ".join(generate("the")))

P(next | 'the'): {'cat': 2, 'dog': 1}
generated: the cat sat the cat sat

After the, the corpus saw cat twice and dog once, so P(cat | the) = 2/3.

Generation just walks this Markov chain. Notice that the output immediately loops (the cat sat the cat sat). That repetition is the visible symptom of the model’s core weakness.

Two problems: sparsity and short context

n-gram models fail in two ways:

• Sparsity. Most possible n-grams never appear in any corpus, so their count — and probability — is zero. The model assigns probability 0 to any sentence containing one. The fix is smoothing: shave probability mass off seen n-grams and give it to unseen ones. Common methods include:

  • add-k
  • backoff to shorter n-grams
  • Kneser-Ney

  The vocabulary explosion is brutal: a trigram model over a 50k vocabulary has 50,000³ possible entries.

• Short context. A bigram sees one word back. Even a 5-gram sees four. Anything longer-range — agreement across a clause, a topic set up sentences ago — is invisible, which is exactly why the generated text loops and drifts. You can’t fix this by raising n: sparsity explodes faster than context grows.

Language models are scored by perplexity — roughly, how surprised the model is by real text (2 raised to the per-word cross-entropy). Lower is better. A perplexity of 100 means the model is as unsure as if choosing uniformly among 100 words at each step.

In one breath

• A language model predicts the next word — the oldest NLP task and the exact objective GPT trains on. Only the machinery changed.

• The n-gram model makes the Markov assumption (next word depends on the last n−1), so it lets you count P(next | prev) from a corpus and generate by walking the chain. The demo loops the cat sat….

• Sparsity: unseen n-grams get probability 0, fixed by smoothing. Its common methods include:

  • add-k
  • backoff
  • Kneser-Ney

  The n-gram space also explodes (50k³ trigrams).

• Short context: an n-gram sees only n−1 words back, so long-range structure is invisible. You can’t just raise n, because sparsity explodes. LMs are scored by perplexity (lower = less surprised).

• RNN LMs learn dense representations with unbounded context (better), and transformers fixed long-range. But n-gram → RNN → GPT is one objective — the architectures improved, along with the amount of context they can use.

Quick check

Next

The objective here, scaled up with a better architecture, is GPT pretraining.

The leap past counting is the RNN/LSTM language model, and long-range modeling is the transformer. The chain-walking rests on Markov chains.