What is tokenization in NLP and why does it matter?

How to think about it

When a support model receives The battery is overheating after the firmware update, it cannot feed that character string directly into its neural layers. Tokenization converts text into a sequence of tokens and integer IDs that the model can turn into vectors; the strategy matters because it determines what counts toward context, how rare words are represented, and how much memory, latency, and cost the request needs.

The mechanism interviewers want

A neural network consumes numbers, not words. A tokenizer owns a fixed vocabulary, meaning a set of known pieces such as words, word fragments, punctuation marks, or bytes. It maps each piece to an integer ID. An embedding table then maps each ID to a learned vector.

The ID itself has no meaning. ID 17 is not inherently “positive” or “a verb”. It means whatever the model learned at row 17 of its embedding table. That is why the tokenizer and model are a matched pair. If you use a different vocabulary, the same ID can point to a completely different vector.

Tokenization usually includes more than splitting. It may normalize the text, meaning clean or standardize it, by lowercasing, changing Unicode forms, or removing accents. It may also add special tokens, which are control markers such as BERT’s [CLS], [SEP], or a model’s end-of-sequence marker.

So the actual pipeline is:

raw text → normalization → token pieces → integer IDs → embeddings

The model processes the IDs, not the original spelling. This is the core reason tokenization affects both behavior and infrastructure.

A concrete example

Here is a BERT tokenizer applied to a short sentence:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
pieces = tokenizer.tokenize("Tokenization matters a lot")

print(pieces)
# ['token', '##ization', 'matters', 'a', 'lot']

The tokenizer splits Tokenization into token and ##ization. In BERT-style WordPiece tokenization, ## means that the piece continues a previous word. It is a tokenizer convention, not a general property of every NLP model.

Because this is bert-base-uncased, the capital T is normalized to lowercase. A call that prepares input for the model may also add special tokens and return IDs, an attention mask, and other fields. The tokenize method above returns pieces only.

The model does not receive the strings token and ##ization. It receives their numeric IDs. Those IDs are then looked up in the embedding table.

This distinction matters when comparing models. The same sentence can produce different token boundaries and different token counts under BERT, GPT-style byte-level BPE, or a tokenizer designed for another language.

The main tokenization strategies

An out-of-vocabulary word, usually called an OOV word, is a word the tokenizer cannot represent using its fixed vocabulary.

Strategy	unhappiness might become	Main trade-off
Word-level	One word, or [UNK] if absent	Short sequences, but a large vocabulary and fragile unknown-word handling
Character-level	Eleven character tokens	Almost no unknown problem, but very long sequences
Subword-level	Pieces such as un and ##happiness	Reuses fragments while keeping sequences manageable
Byte-based	UTF-8 byte pieces, possibly merged	Can represent arbitrary text, but unusual text may take many tokens

Word-level tokenization is easy to explain and sometimes useful in small, controlled systems. But play, plays, played, and playing may receive four unrelated IDs. If playfully was absent from the training vocabulary, the tokenizer may replace it with [UNK], losing its spelling and morphology.

Character tokenization avoids that problem because any character in the supported alphabet can be represented. The price is sequence length. The eleven characters in unhappiness require eleven processing positions instead of one or two.

Subword tokenization tries to sit between those extremes. Byte-pair encoding, or BPE, starts with small symbols such as characters or bytes and repeatedly merges frequent adjacent pairs. WordPiece also learns a subword vocabulary, but uses a different training objective and commonly selects the longest valid pieces during segmentation. Neither method guarantees that a piece is a real linguistic morpheme. The split is learned from text statistics, not from a linguist with a red pen.

Why the choice matters in a real system

It controls context length and compute. A token is not necessarily a word. A rough English planning rule is one token for about four characters or three-quarters of a word, but this varies by tokenizer, language, punctuation, code, and formatting.

A 4,000-word document might occupy roughly 5,300 tokens. In an 8,192-token context window, that leaves only about 2,900 positions for system instructions, conversation history, and the answer. The exact count must come from the model’s tokenizer, not from splitting on spaces.

This also affects latency. In full attention, each token can compare with every other token, so the number of pairwise interactions grows roughly quadratically with sequence length. Doubling a prompt from 4,000 to 8,000 tokens is not merely twice as much work for that part of the model. In a batch, one unusually long request can also force shorter requests to be padded to the same length.

It controls vocabulary cost. Suppose an embedding has width 768. A vocabulary of 30,000 tokens requires 30,000 × 768 = 23,040,000 parameters. A vocabulary of 100,000 requires 76,800,000 parameters. At half precision, those embedding matrices occupy roughly 46 MB and 154 MB respectively, before gradients, optimizer state, or an untied output head.

A larger vocabulary can shorten sequences, but it makes the embedding and output layers larger. Rare vocabulary entries also receive fewer training examples. A smaller vocabulary reduces those costs but represents text using more pieces.

It affects generalization. Subwords let a model reuse pieces across related words. A model can encounter replaying and recognize a piece related to play, even if it never saw that exact full word during training. That does not mean the tokenizer understands the word’s meaning. It only gives the model reusable symbols. The model still has to learn the relationship from data.

It affects languages and domains unevenly. A tokenizer trained mainly on English may split Hindi, Arabic, Thai, source-code identifiers, log lines, or emoji into many more tokens than an English sentence of similar visual length. That raises cost and makes context fill faster. It can also make learning harder because one meaningful unit is spread across many positions.

A useful production check is therefore not just average token count. Measure token lengths on the actual traffic: English and non-English text, product IDs, punctuation-heavy tickets, JSON, code, and emoji.

The senior-level nuance

The common answer “subword tokenization eliminates OOV tokens” is too strong.

BERT’s WordPiece tokenizer can still emit [UNK] when it cannot find a valid decomposition. Byte-level tokenizers or tokenizers with byte fallback can represent arbitrary Unicode text, but they may do so inefficiently. COVID-19 might become pieces resembling covid, -, and 19 in one tokenizer, while another tokenizer chooses a different segmentation. Never promise a specific split without checking the actual tokenizer.

The tokenizer is also part of the model artifact. When loading a pretrained checkpoint, use the tokenizer shipped for that checkpoint and pin its vocabulary, merge rules, normalization settings, and revision. Changing tokenization changes the mapping from text to IDs. The model has not learned the new mapping unless it is retrained or specifically adapted for it.

If training from scratch, there is no universally best vocabulary size. Compare compression, downstream quality, multilingual coverage, encoding speed, and model size. A tokenizer that saves 8 percent of English tokens may be a poor choice if it doubles token counts for the languages your customers actually use.

There are cases where a general subword tokenizer is the wrong tool. A short classifier for a fixed set of SKU strings may work better with character features or a direct lookup. A keyword filter may only need carefully defined normalization and whitespace rules. For an existing large language model, however, replacing the tokenizer is usually not a sensible optimization; it breaks the model’s learned input space.

A common failure mode is tokenizer-model mismatch. The first symptom may be a sudden jump in p95 token length, a spike in [UNK], degraded predictions, or an index-out-of-range error when an input ID exceeds the model’s embedding table. Another is silent truncation: the support bot answers fluently but ignores the account number or the final paragraph because the useful text was cut to fit the context limit.

For token-classification tasks such as named-entity recognition, there is an additional trap. One word can become several subwords, so word-level labels and character offsets must be aligned to the resulting pieces. A bad alignment can produce plausible-looking output with every entity span shifted.

What they’ll ask next

How do BPE and WordPiece differ?
Both learn reusable subword pieces. BPE usually merges the most frequent adjacent symbols, while WordPiece uses a different vocabulary-learning score and commonly applies longest-match-first segmentation. Their outputs, unknown-token behavior, and special conventions differ, so they are not interchangeable.

Why not use character tokens everywhere and avoid unknown words?
Character tokens handle arbitrary spelling and noisy text, but they create much longer sequences. Longer sequences increase attention work and make the model spend more steps reconstructing patterns that a useful subword could represent in one piece. Character tokenization can still win for short, noisy strings such as usernames or misspellings.

How would you debug tokenization in production?
Log token counts and truncation rates, sample the actual pieces for Unicode, punctuation, code, and product IDs, and verify that training and serving use the same tokenizer files. Monitor p50 and p95 lengths rather than relying on word counts.

Say this in the interview

“Tokenization is the model-specific conversion from text to tokens and integer IDs; subwords usually balance vocabulary size and sequence length, so the tokenizer determines context usage, cost, unknown-word behavior, and how well the model handles rare words.”