Tokenization & BPE

Before a transformer can do anything with your text, the text has to become numbers. A model never sees the word “running” — it sees something like token 12973, looks that ID up in an embedding table, and gets a vector. The rule that turns text into those IDs is tokenization. It quietly explains a surprising amount:

• why API calls cost what they cost
• why your context window fills faster in some languages
• why a frontier model can write an essay but can’t count the letters in “strawberry.”

Why not just words, or just characters?

Two obvious choices both fail:

• One token per word → the vocabulary is unbounded (typos, names, new words, every language) and the model has no idea that “run”, “running”, and “runner” are related.
• One token per character → tiny vocabulary, but sequences become enormous, and attention is quadratic in length — far too expensive.

The winning compromise is subwords. It handles input in three ways:

• Common words stay whole (“the”, “model”).
• Rare words split into reusable pieces (“token” + “ization”).
• Spaces and digits fragment differently too.

That keeps the vocabulary fixed (~50k–200k tokens) while handling any input:

Byte-pair encoding, the algorithm

The dominant method, BPE, learns its vocabulary from data with one beautifully simple loop. It starts from individual characters (or bytes), then repeatedly finds the most frequent adjacent pair and merges it into a new token. Do that a few thousand times, and frequent sequences naturally become single tokens.

Build a tiny BPE trainer and watch merges form on a toy corpus:

from collections import Counter

# A toy corpus, pre-split into words shown as character sequences with a marker.
corpus = ["low", "low", "low", "lowest", "lowest", "newer", "newer", "wider"]
# represent each word as a list of chars + an end-of-word marker
words = [list(w) + ["_"] for w in corpus]

def most_frequent_pair(words):
    pairs = Counter()
    for w in words:
        for i in range(len(w) - 1):
            pairs[(w[i], w[i+1])] += 1
    return pairs.most_common(1)[0] if pairs else (None, 0)

def merge(words, pair):
    a, b = pair
    out = []
    for w in words:
        i, new = 0, []
        while i < len(w):
            if i < len(w)-1 and w[i] == a and w[i+1] == b:
                new.append(a + b); i += 2          # merge into one token
            else:
                new.append(w[i]); i += 1
        out.append(new)
    return out

# Run a few merge steps
for step in range(5):
    pair, count = most_frequent_pair(words)
    words = merge(words, pair)
    print(f"merge {step+1}: {pair[0]!r}+{pair[1]!r} (seen {count}x)  ->  new token {pair[0]+pair[1]!r}")

print("\nfinal tokenization of 'lowest':", words[3])

merge 1: 'l'+'o' (seen 5x)  ->  new token 'lo'
merge 2: 'lo'+'w' (seen 5x)  ->  new token 'low'
merge 3: 'low'+'_' (seen 3x)  ->  new token 'low_'
merge 4: 'e'+'r' (seen 3x)  ->  new token 'er'
merge 5: 'er'+'_' (seen 3x)  ->  new token 'er_'

final tokenization of 'lowest': ['low', 'e', 's', 't', '_']

That’s the whole idea behind GPT’s tiktoken, SentencePiece, and friends — real tokenizers run this merge process over billions of characters and store the result as a vocabulary plus a merge table.

Token → embedding

Once text is a list of token IDs, the model’s embedding table (a big lookup matrix of shape vocab_size × d_model) maps each ID to a vector. That vector is the actual input to self-attention. The input and output embeddings are often tied (shared weights) to save parameters.

import torch.nn as nn

embed = nn.Embedding(num_embeddings=50257, embedding_dim=768)  # GPT-2 sizes
ids = tokenizer.encode("hello world")    # → [15496, 995]
vectors = embed(torch.tensor(ids))       # → shape (2, 768)

In one breath

• A model never sees letters or words — text becomes token IDs, each looked up in an embedding table to get a vector.
• Words give an unbounded vocabulary; characters make sequences too long (attention is quadratic). Subwords are the compromise: common words whole, rare ones split into reusable pieces, fixed ~50k–200k vocabulary.
• BPE builds that vocabulary by repeatedly merging the most frequent adjacent pair, starting from characters/bytes.
• Tokenization sets cost and context (you pay per token).
• It penalizes non-English text (2–4× more tokens).
• It hides characters inside tokens — which is why an LLM can’t count the r’s in “strawberry.”
• Always measure tokens with the model’s own tokenizer, never characters; ~4 chars/token in English is only a rough rule that code and other languages break.

Quick check

Next

Now that text is a sequence of embedded tokens, we can do the thing transformers are famous for: let every token look at every other token, with self-attention.