Tokenization

When an LLM “reads” your prompt, it does not see characters or words. It sees tokens — the fixed units of text a model’s vocabulary is built from, roughly word-parts rather than whole words. And once you know that tokens are the unit, a lot of production friction stops being mysterious: pricing is per token, context windows are measured in tokens, rate limits are in tokens. This lesson is about making that unit visible and countable before you send anything.

Why sub-word tokens

Why not just use characters, or whole words? Each extreme fails in its own way. A character vocabulary is tiny but far too coarse — every word becomes five or more steps, and the model has to re-learn from scratch that "running" and "runs" share a stem. A word vocabulary is the opposite: it balloons to millions of entries and still cannot handle a typo or a word it has never seen. The compromise that won is sub-word tokenization — common words stay one token, while rare or compound words split into pieces:

So "the" is one token, "running" is usually one, "datarekha" is two or three, and non-Latin scripts often expand further. Two algorithms dominate: Byte-Pair Encoding (BPE) — what OpenAI’s tiktoken uses, starting from bytes and repeatedly merging the most common adjacent pair until it hits the target vocabulary size — and SentencePiece, the Google/Meta favourite, which is language-agnostic and treats whitespace as just another character. You will not implement either; you just need to know they exist and to count tokens before sending big payloads.

Counting tokens with tiktoken

tiktoken is OpenAI’s tokenizer library, and it is the canonical way to know a string’s token count before you spend money on it:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")   # GPT-4 / GPT-3.5 family
tokens = enc.encode("the quick brown fox jumps over the lazy dog")
print(len(tokens))         # 9 — nine common words, one token each
print(enc.decode(tokens))  # round-trips back to the original

To make the mechanics runnable here without tiktoken, the toy tokenizer below approximates a sub-word split — its counts are not the real ones, but the API shape is exactly what you would ship:

import re

class ToyTokenizer:
    """A toy sub-word splitter — real models use BPE/SentencePiece.
    Short chunks are one token; long words break into ~4-char pieces."""
    def tokens(self, text):
        out = []
        for word in re.findall(r"\S+|\s+", text):
            if word.isspace() or len(word) <= 4:
                out.append(word)
            else:
                out += [word[i:i + 4] for i in range(0, len(word), 4)]
        return out

    def count(self, text):
        return len(self.tokens(text))

t = ToyTokenizer()
samples = ["the", "the quick brown fox", "datarekha",
           "supercalifragilisticexpialidocious", "Hello, world! How are you today?"]
for s in samples:
    print(f"{t.count(s):>3} tokens / {len(s):>3} chars  | {s}")

  1 tokens /   3 chars  | the
  9 tokens /  19 chars  | the quick brown fox
  3 tokens /   9 chars  | datarekha
  9 tokens /  34 chars  | supercalifragilisticexpialidocious
 14 tokens /  32 chars  | Hello, world! How are you today?

The exact counts vary by real tokenizer (cl100k_base, o200k_base, SentencePiece for Llama, and so on), but the rule of thumb for English is roughly 4 characters per token, or about 0.75 tokens per word — not one token per word. Code, JSON, and non-Latin scripts run more expensive, sometimes two to three times.

Cost is tokens times price

Here is a back-of-the-envelope you will redo a hundred times in your career, with the input and output costs broken out:

def estimate(prompt_chars, completion_chars, in_price_1m, out_price_1m):
    in_tok = prompt_chars / 4          # ~4 chars per token in English
    out_tok = completion_chars / 4
    in_cost = (in_tok / 1_000_000) * in_price_1m
    out_cost = (out_tok / 1_000_000) * out_price_1m
    return in_tok, out_tok, in_cost, out_cost

# 1M reviews: ~500 input chars, ~50 output chars each. Hypothetical 2026 pricing.
in_tok, out_tok, in_cost, out_cost = estimate(
    1_000_000 * 500, 1_000_000 * 50,
    in_price_1m=1.25, out_price_1m=10.0,
)
print(f"Input tokens:  {in_tok:>14,.0f}  (${in_cost:,.2f})")
print(f"Output tokens: {out_tok:>14,.0f}  (${out_cost:,.2f})")
print(f"Estimated total: ${in_cost + out_cost:,.2f}")

Input tokens:     125,000,000  ($156.25)
Output tokens:     12,500,000  ($125.00)
Estimated total: $281.25

Read the split carefully, because it overturns a common myth. Output tokens are priced about 8× higher (10.00 versus 1.25 per million here), yet input still edges out output on this bill — $156.25 to $125.00 — because there is 10× more input volume than output. The real rule is a ratio: output dominates the bill once your output volume exceeds about one-eighth of your input volume. Here output is one-tenth of input, just under that line. Shift to chatty, free-form completions and output runs away with the cost in a hurry.

Context windows are in tokens

Every model has a maximum combined token budget for input plus output — GPT-4-turbo was 128k, Claude Opus 4 reached 200k, and million-token models now exist (at a price that scales accordingly). You can blow the window three ways, each with its own fix: the system-plus-user prompt is too long on its own (a giant code dump or full PDF); the history accumulates across a multi-turn chat until it overflows; or the retrieved context pasted in for RAG is too verbose.

Truncation strategies

You will hit the window, and the naive fix — chop the oldest characters — almost always discards the most important information, like the system prompt or the user’s original question. The better pattern is to anchor what is required and slide a window over the rest:

def count_tokens(s):
    return max(1, len(s) // 4)        # a rough char-based proxy

history = [
    {"role": "system",    "content": "You are a helpful assistant. " * 50},
    {"role": "user",      "content": "What is the capital of France?"},
    {"role": "assistant", "content": "Paris."},
    {"role": "user",      "content": "Tell me about it." + " details" * 200},
    {"role": "assistant", "content": "Paris is the capital of France..." * 50},
    {"role": "user",      "content": "What's the population?"},
]

BUDGET = 800

def fit(messages, budget):
    system = [m for m in messages if m["role"] == "system"]   # always keep
    rest = [m for m in messages if m["role"] != "system"]
    latest = rest[-1:]                                          # always keep the last user turn
    middle = rest[:-1]
    used = sum(count_tokens(m["content"]) for m in system + latest)
    kept = []
    for m in reversed(middle):                                 # newest middle first
        t = count_tokens(m["content"])
        if used + t > budget:
            break
        kept.insert(0, m)
        used += t
    return system + kept + latest, used

trimmed, used = fit(history, BUDGET)
print(f"Original messages: {len(history)} -> kept: {len(trimmed)}")
print(f"Tokens used: ~{used} / {BUDGET}")
for m in trimmed:
    print(f"  [{m['role']:>9}] {m['content'][:60]!r}...")

Original messages: 6 -> kept: 3
Tokens used: ~779 / 800
  [   system] 'You are a helpful assistant. You are a helpful assistant. Yo'...
  [assistant] 'Paris is the capital of France...Paris is the capital of Fra'...
  [     user] "What's the population?"...

The system prompt and the latest user turn were anchored; then the newest middle message that fit (the long assistant reply) was kept, and the older turns were dropped — landing at 779 of the 800-token budget. Most chat apps keep the system prompt plus the last six-to-ten turns; RAG systems keep the system prompt, the retrieved chunks, and the current question, in that priority order.

In one breath

• Models read sub-word tokens, not words or characters — common words are one token, rare ones split.
• Count before you send: tiktoken for OpenAI models; English is ~4 chars per token.
• Output is priced ~8× input per token, but the bill depends on the volume ratio — measure both.
• Context windows are a combined input+output token budget; you can overflow via prompt, history, or RAG.
• Truncate by anchoring what is required (system prompt, latest question) and sliding a window over the rest.

Practice

What’s next

Now that you can predict the input cost, the next lesson is making the actual calls — the modern OpenAI and Anthropic SDKs, side by side.