Tokenization & BPE

Before a transformer can do anything with your text, the text has to become numbers. A model never sees the word “running” — it sees something like token 12973, looks that ID up in an embedding table, and gets a vector. The rule that turns text into those IDs is tokenization, and it quietly explains a surprising amount: why API calls cost what they cost, why your context window fills faster in some languages, and why a frontier model can write an essay but can’t count the letters in “strawberry.”

Why not just words, or just characters?

Two obvious choices both fail:

• One token per word → the vocabulary is unbounded (typos, names, new words, every language) and the model has no idea that “run”, “running”, and “runner” are related.
• One token per character → tiny vocabulary, but sequences become enormous, and attention is quadratic in length — far too expensive.

The winning compromise is subwords: common words stay whole (“the”, “model”), rare words split into reusable pieces (“token” + “ization”). That keeps the vocabulary fixed (~50k–200k tokens) while handling any input — and spaces, digits, and rare words all fragment differently:

Byte-pair encoding, the algorithm

The dominant method, BPE, learns its vocabulary from data with one beautifully simple loop: start from individual characters (or bytes), then repeatedly find the most frequent adjacent pair and merge it into a new token. Do that a few thousand times and frequent sequences naturally become single tokens.

Build a tiny BPE trainer and watch merges form on a toy corpus:

from collections import Counter

# A toy corpus, pre-split into words shown as character sequences with a marker.
corpus = ["low", "low", "low", "lowest", "lowest", "newer", "newer", "wider"]
# represent each word as a list of chars + an end-of-word marker
words = [list(w) + ["_"] for w in corpus]

def most_frequent_pair(words):
    pairs = Counter()
    for w in words:
        for i in range(len(w) - 1):
            pairs[(w[i], w[i+1])] += 1
    return pairs.most_common(1)[0] if pairs else (None, 0)

def merge(words, pair):
    a, b = pair
    out = []
    for w in words:
        i, new = 0, []
        while i < len(w):
            if i < len(w)-1 and w[i] == a and w[i+1] == b:
                new.append(a + b); i += 2          # merge into one token
            else:
                new.append(w[i]); i += 1
        out.append(new)
    return out

# Run a few merge steps
for step in range(5):
    pair, count = most_frequent_pair(words)
    words = merge(words, pair)
    print(f"merge {step+1}: {pair[0]!r}+{pair[1]!r} (seen {count}x)  ->  new token {pair[0]+pair[1]!r}")

print("\nfinal tokenization of 'lowest':", words[3])

merge 1: 'l'+'o' (seen 5x)  ->  new token 'lo'
merge 2: 'lo'+'w' (seen 5x)  ->  new token 'low'
merge 3: 'low'+'_' (seen 3x)  ->  new token 'low_'
merge 4: 'e'+'r' (seen 3x)  ->  new token 'er'
merge 5: 'er'+'_' (seen 3x)  ->  new token 'er_'

final tokenization of 'lowest': ['low', 'e', 's', 't', '_']

That’s the whole idea behind GPT’s tiktoken, SentencePiece, and friends — real tokenizers run this merge process over billions of characters and store the result as a vocabulary plus a merge table.

Token → embedding

Once text is a list of token IDs, the model’s embedding table (a big lookup matrix of shape vocab_size × d_model) maps each ID to a vector. That vector is the actual input to self-attention. The input and output embeddings are often tied (shared weights) to save parameters.

import torch.nn as nn
embed = nn.Embedding(num_embeddings=50257, embedding_dim=768)  # GPT-2 sizes
ids = tokenizer.encode("hello world")    # → [15496, 995]
vectors = embed(torch.tensor(ids))       # → shape (2, 768)

In one breath

• A model never sees letters or words — text becomes token IDs, each looked up in an embedding table to get a vector.
• Words give an unbounded vocabulary; characters make sequences too long (attention is quadratic). Subwords are the compromise: common words whole, rare ones split into reusable pieces, fixed ~50k–200k vocabulary.
• BPE builds that vocabulary by repeatedly merging the most frequent adjacent pair, starting from characters/bytes.
• Tokenization sets cost and context (you pay per token), penalizes non-English text (2–4× more tokens), and hides characters inside tokens — which is why an LLM can’t count the r’s in “strawberry.”
• Always measure tokens with the model’s own tokenizer, never characters; ~4 chars/token in English is only a rough rule that code and other languages break.

Quick check

Next

Now that text is a sequence of embedded tokens, we can do the thing transformers are famous for: let every token look at every other token, with self-attention.