How does tokenization work, and why do LLMs rely on subword tokenizers like BPE?

How to think about it

Tokenization is the preprocessing step that converts text into a sequence of integer IDs that a language model can process. LLMs use subword tokenizers such as Byte-Pair Encoding, or BPE, because they provide a practical middle ground: common text becomes compact, while rare words, names, misspellings, and new words can still be represented without an unknown-word failure.

Why tokenization exists

A transformer does not receive the string The server is down. It receives something like 101, 582, 19, 407, although the actual IDs depend entirely on the model’s tokenizer.

Those IDs are indexes, not meanings. The model uses each ID to look up a learned vector in an embedding table. The transformer processes the vector sequence and predicts the next token ID. A decoder then turns the predicted IDs back into text.

That distinction matters. A token is not necessarily a word, and tokenization is not language understanding. A token might be a whole word, part of a word, punctuation, or a fragment that includes surrounding whitespace. In many BPE tokenizers, the token for hello with a leading space is different from the token for hello without one.

A tokenizer also handles special control tokens used by a model, such as markers for the beginning of a sequence, the end of a sequence, or a user and assistant turn. These markers have reserved IDs and must be handled consistently. They are part of the model’s input format, not decorative punctuation.

How BPE works

BPE learns a vocabulary from a training corpus. The basic procedure is:

1. Start with small base units, usually characters or bytes.
2. Count adjacent pairs of units across the corpus.
3. Merge the most frequent pair into a new unit.
4. Repeat until the vocabulary reaches its target size.

The learned merge rules are saved. At inference time, the tokenizer applies those rules to new text, producing the longest or highest-priority pieces allowed by the learned vocabulary.

Modern tokenizers often have an additional pre-tokenization step before BPE. It may split text around whitespace, punctuation, or language-specific patterns. Byte-level BPE starts from byte values, which means it can represent arbitrary valid UTF-8 text without needing a separate unknown token. Other BPE variants start from characters or from pre-split word pieces.

Here is a small toy corpus:

low low lower lowest

Suppose a word boundary is shown with the marker |. The initial sequences are:

l o w |       l o w |       l o w e r |       l o w e s t |

The most frequent adjacent pairs are:

Pair	Count
l o	4
o w	4
w |	2
w e	2
e r	1
e s	1

Assume the deterministic tie-breaker chooses l o. BPE replaces every occurrence with lo:

lo w |       lo w |       lo w e r |       lo w e s t |

Now lo w occurs four times, so it can become low. The tokenizer has discovered a useful recurring fragment without being told anything about English morphology.

With later merges, low might become one token, while lower could be represented as low plus smaller pieces. A new string such as lowering can reuse low and encode the rest from other known pieces. The exact split depends on the trained vocabulary and merge order; BPE does not promise that it will identify linguistically perfect roots and suffixes.

That last point is a common interview trap.

Common misconception: BPE is not a linguistic analyzer. It finds statistically useful character or byte sequences. A piece that looks like a suffix may have been learned simply because it appears often.

The integer assigned to each piece is arbitrary. Token ID 5,000 does not mean “verb” or “important word.” The model learns what an ID represents through training.

Why subwords are the compromise

A word-level tokenizer sounds simple: assign one token to every word. It fails in practice.

A vocabulary containing every useful word, name, product code, spelling variant, and inflected form would be enormous. Rare words would still appear after training. A word-level tokenizer must then map an unseen word to a generic unknown token, losing its internal spelling.

A character- or byte-level tokenizer has the opposite problem. It can represent any input, but a sentence becomes much longer. Longer sequences increase memory use and latency, and full self-attention becomes more expensive as sequence length grows.

Subwords sit between those extremes:

• Frequent strings can be represented by one token.
• Rare strings can be decomposed into smaller known pieces.
• The vocabulary remains manageable.
• The model sees fewer positions than it would with individual characters or bytes.

For example, a tokenizer may encode a common word such as international compactly, while breaking an unfamiliar identifier such as ZXQ-4817-alpha into several pieces. That is useful because the model can still see recurring fragments such as digits, punctuation, or alpha rather than treating the entire identifier as an indivisible unknown.

Subwords also help with morphology and reuse. walk, walking, and walked may share pieces, allowing statistical information to transfer between them. But this is a consequence of the learned vocabulary, not a guarantee.

The senior-level trade-off

The tokenizer vocabulary size creates a real engineering trade-off.

A larger vocabulary can make sequences shorter because more strings have dedicated tokens. That reduces the number of transformer positions. But the embedding table grows with vocabulary size, and the model’s output layer must usually produce a logit for every vocabulary entry. Larger output layers consume memory and computation.

A smaller vocabulary has cheaper vocabulary-dependent layers, but it produces longer sequences. For full self-attention, the pairwise attention score matrix grows roughly with the square of the sequence length. Doubling a sequence from 4,000 to 8,000 tokens makes that matrix about four times larger, all else being equal.

Tokenization therefore affects context limits, latency, and cost. Consider a support chatbot with an 8,192-token context limit:

• System instructions: 900 tokens
• Conversation history: 4,800 tokens
• Retrieved documentation: 1,400 tokens
• Reserved space for the answer: 800 tokens

The total is 7,900 tokens, leaving only 292 tokens of headroom. A 500-token tool result pushes the request to 8,400 tokens. The system must truncate content, reject the request, or produce a shorter answer.

Counting characters would not reliably prevent this. A string’s token count depends on the exact tokenizer, language, whitespace, punctuation, Unicode representation, and even whether special tokens are included. Production systems should count with the tokenizer paired with the deployed model and reserve space for the output.

Tokenization is also uneven across languages and domains. A tokenizer trained mostly on English may use more tokens for the same idea expressed in another language. Source code, mathematical notation, URLs, and long numeric strings can also fragment awkwardly. The fairest comparison is not “one token equals one word”; it is how many tokens the tokenizer needs for the workloads the model will actually serve.

Many LLMs use BPE or a close relative, but BPE is not universal. WordPiece, common in BERT-style models, learns pieces with a different scoring objective. SentencePiece is a tokenizer framework that can train BPE or Unigram models and can operate directly on raw text rather than requiring whitespace-separated words. The goal is similar, but the merge or segmentation procedure differs.

Finally, the tokenizer is part of the model. If a model was trained with one vocabulary and merge table, swapping in another tokenizer is not a harmless preprocessing change. The same integer may now refer to a different string, so the embedding lookup becomes semantically wrong. At minimum, the input embeddings and output vocabulary would need to be adapted, and in practice the model usually needs substantial retraining.

A failure mode I would watch for

The first symptom of a tokenizer mismatch is often surprising prompt accounting: the application logs 6,000 input tokens, while the model gateway reports 7,200. Another symptom is fluent-looking but degraded output when IDs remain in range but refer to different pieces. If IDs are incompatible, the failure may be more obvious: an embedding-index error or a rejected request.

I would inspect the normalized text, the individual pieces, the integer IDs, special-token handling, and the final count. I would pin the tokenizer files and version, test spaces and Unicode text, and verify that encoding followed by decoding preserves the intended text. The tokenizer used for budgeting, training, evaluation, and serving must be the same model-specific artifact.

What they’ll ask next

Why not use one token per character?

Character tokenization avoids unknown words, but it creates much longer sequences. Longer sequences cost more memory and attention computation. Subword tokenization retains the robustness of small units while compressing frequent patterns.

Is BPE the same as SentencePiece?

No. BPE is a vocabulary-learning algorithm based on repeated pair merges. SentencePiece is a toolkit and model family that can implement BPE or alternatives such as Unigram segmentation. WordPiece is another related approach with a different training criterion.

What happens to a word the tokenizer has never seen?

It is split into pieces already present in the vocabulary. With a byte-level tokenizer, any valid UTF-8 input can ultimately be represented as bytes. The model does not need a dedicated token for every possible word, although unusual strings may require many tokens.

Say this in the interview

“Tokenization maps text to model-specific IDs, and BPE learns frequent character or byte merges so common text stays compact while rare or new strings remain representable; the trade-off is vocabulary size versus sequence length, which directly affects memory, latency, and context usage.”