Tokens, chat templates and budgets

At 3:07 a.m., your support assistant starts returning answers that end halfway through a JSON object.

The prompt looks harmless. A short system instruction. Five retrieved documentation chunks. A fourteen-turn conversation. The request is about 8,000 characters, so someone assumes it should fit comfortably. The API instead returns context_length_exceeded, or accepts the request and stops the answer after "owner": "on-call.

The next day, an engineer deletes two paragraphs from the prompt. The bug returns when a German customer pastes a stack trace, or when the conversation reaches its twentieth turn.

The missing measurement is not characters. It is tokens: the model’s actual units of text. Tokens determine how much context fits, how much input and output cost, and how long generation takes.

A token is not a word

A token is an integer ID representing a small piece of text in a model’s vocabulary. It might be a whole word, part of a word, punctuation, whitespace attached to a word, or part of a number.

Modern models usually use subword tokenization. They store frequent pieces such as ing, tion, or http, then combine smaller pieces to represent unfamiliar text. The tokenizer converts text into token IDs before the model sees it; the model receives IDs, not the original string.

• A word is a linguistic unit.
• A token is a model-specific encoding unit.
• A character is a written symbol.
• A byte is a low-level text representation.

There is no universal conversion rate. For a common GPT-style tokenizer, these are illustrations rather than promises:

Text	What the tokenizer sees	Approximate lesson
The cat sat on the mat.	Frequent words plus punctuation	About 7 tokens
def add(a, b): return a + b	Keywords, identifiers, spaces, punctuation	Roughly 12 tokens
東京に住んでいます。	More small pieces for many characters	It may take 10–12 tokens
97812345678901234567890	An unfamiliar digit sequence split into pieces	Often 8–15 tokens

A rough English estimate is four characters per token, but it is unreliable for code, tables, emoji, URLs, numbers, and other languages. Frequent text such as the may be one piece; a random account number usually is not. Count with the tokenizer for the model you will actually call.

This affects both context fit and price: a multilingual application can hit its limit, or cost more, for one language even when character counts match. The practical rule is to budget in tokens, not characters.

The chat template is part of the prompt

A chat interface gives you role-labelled messages:

messages = [
    {"role": "system", "content": "Answer with short, accurate steps."},
    {"role": "user", "content": "How do I rotate an API key?"},
]

A language model receives one token sequence, not a native list. A chat template serializes the messages into that sequence:

[system marker]
Answer with short, accurate steps.
[end marker]
[user marker]
How do I rotate an API key?
[end marker]
[assistant marker]

Marker names differ by model. These special tokens were present during instruction tuning, teaching the model where messages begin and end and where the assistant should answer. They count toward the context window even when they are not visible in the message text.

Common mistakes include using another model’s template, formatting messages twice, inserting special tokens twice, omitting the assistant-generation marker, or sending chat formatting to a base completion model. Symptoms include role labels in the answer, ignored system instructions, generic continuation, or fluent nonsense.

Hugging Face Transformers’ apply_chat_template uses the tokenizer’s stored template and can return token IDs directly. That prevents visible message text from being counted without its structural tokens.

Input and output behave differently

A request has two budgets:

• Input tokens: the formatted prompt, including instructions, retrieved text, history, the question, and template markers.
• Output tokens: the new tokens the model may generate.

For an 8,192-token context window:

input tokens + reserved output tokens <= 8,192

If you allow up to 1,200 output tokens, the request must leave room for all 1,200, even if the model usually stops sooner.

The two budgets affect latency differently. During prefill, the model processes the existing prompt; this mostly determines time to first token (TTFT). During decode, it generates one token after another, so a long answer keeps the connection busy longer. Input and output prices also commonly differ. Use token counts and the provider’s current rates rather than word count.

A worked RAG budget

Suppose a support assistant uses a model with an 8,192-token context window. The application counts the fully formatted prompt:

Component	Measured tokens
System instructions	310
Five retrieved chunks	2,100
Conversation history	2,460
Current question	44
Template and special tokens	86
Input total	5,000
Reserved answer	1,200
Worst-case request	6,200

There are 1,992 tokens of headroom, so the request fits. The input is 5,000 tokens, not 4,914: the final serialized sequence includes 86 tokens of template overhead.

After several turns, history grows to 4,800 tokens:

310 + 2,100 + 4,800 + 44 + 86 = 7,340 input tokens

With the same reservation:

7,340 + 1,200 = 8,540

The request is 348 tokens over the limit. The problem is the budget, not retrieval.

At hypothetical rates of $3 per million input tokens and $15 per million output tokens, the first request’s maximum cost is:

• Input: 5,000 / 1,000,000 × $3 = $0.015
• Reserved output: 1,200 / 1,000,000 × $15 = $0.018
• Maximum total: $0.033

This is a planning ceiling. Provider definitions of maximum output may include hidden reasoning tokens. Those tokens can consume context and output budget, increase latency, and truncate the visible answer, so follow the provider’s billing and limit documentation.

The production sequence is:

1. Format retrieved documents and select or summarize history.
2. Add the question.
3. Apply the exact chat template.
4. Count the resulting token IDs.
5. Compare the count plus output reservation with the real context limit.
6. Send only if it fits.

Account for wrappers, tool definitions, and serving-stack limits; the advertised model maximum is not always the application’s usable budget.

Truncation without amnesia

When a prompt is too large, truncation removes or compresses input. Keeping the system message and newest turns while deleting the middle can silently remove the decision or evidence that the current question refers to. A model has no hidden conversation memory: if your application does not send the evidence, the model cannot use it.

A task-dependent priority order is:

1. Protect non-negotiable system instructions and the current request.
2. Keep required tool calls with their tool results.
3. Protect safety-critical, citation-critical, or otherwise necessary evidence.
4. Rank remaining history and retrieved material by relevance, recency, and cost.
5. Replace older, lower-priority history with a clearly labelled summary when safe.

Strategy	Preserves best	First risk	Use it when
Hard reject	All context	Visible error or retry	Missing context would be unsafe or legally important
Keep newest turns	Recency and tone	Old decisions vanish	Casual, short-lived chat
Rolling summary	Older decisions and facts	Summary drift or omission	Long-running assistants with reviewable summaries
Retrieve old history	Relevant past facts	Exact sequence may disappear	Large support or project conversations

A summary is not automatically trustworthy because it is shorter. Keep original turns when auditability matters, and verify summaries when important facts change. For document-heavy questions, reducing chunk size or retrieving fewer, more relevant chunks may be better than sacrificing recent conversation.

Count before sending

For a local Transformers model, count the exact chat-formatted sequence:

from transformers import AutoTokenizer

model_name = "Qwen/Qwen2.5-7B-Instruct"
application_context_budget = 8192  # Intentional application cap; the checkpoint publishes 32,768 tokens, subject to the serving stack.
reserved_output = 1200

messages = [
    {
        "role": "system",
        "content": "Answer with accurate, concise troubleshooting steps.",
    },
    {
        "role": "user",
        "content": "Why does the service return HTTP 429 after a deployment?",
    },
]

tokenizer = AutoTokenizer.from_pretrained(model_name)

input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
)

input_tokens = len(input_ids)
total_reserved = input_tokens + reserved_output

print(f"input tokens: {input_tokens}")
print(f"input plus reserved output: {total_reserved}")

if total_reserved > application_context_budget:
    raise ValueError(
        f"Prompt budget exceeded: {total_reserved} tokens "
        f"for a {application_context_budget}-token application budget"
    )

add_generation_prompt=True adds the marker telling an instruct model to begin an assistant response. If the server applies the template, count with its tokenizer or token-counting endpoint. For hosted APIs, also count tool schemas, images, and other request fields; a function definition is input too.

Diagnose the failure

Symptom	Likely cause	Fix
context_length_exceeded	Input plus reservation exceeds the limit	Count the final prompt and trim or reduce the reservation
JSON stops halfway through	Generation limit was reached	Reserve more output, shorten input, and check the finish reason
Role labels appear or instructions are ignored	Wrong, duplicated, or missing template markers	Use the checkpoint’s tokenizer template exactly once
An earlier decision is contradicted	Truncation removed the relevant turn	Preserve it, retrieve it, or add a verified summary

The honest limitation is that a token budget guarantees fit, not quality. A prompt can fit inside 8,192 tokens and still bury the relevant fact among irrelevant text. A summary can fit and still be wrong, while a larger context can add cost and latency without improving the answer. Counting prevents accidental overflow; retrieval, prompt design, and evaluation determine whether the remaining context helps.

What to remember

• Tokens are model-specific text pieces, not words, characters, or fixed bytes.
• Code, numbers, URLs, and non-English text can be token-dense.
• A chat template converts messages into the token sequence the model reads; match it to the model.
• Budget input tokens + reserved output against the context window.
• Truncate deliberately, preserving required instructions, evidence, complete tool exchanges, and trustworthy summaries.

Quick check

0/3

Q1

Tokens are counted twice by every modelThe request also needs room for the reserved generated outputThe model converts tokens back into words firstOnly system messages count toward context

Check answer

Q2

Join role names and message text with newline charactersUse the chat template stored with the model's tokenizerUse the template from any model in the same model familySend the messages twice so the model sees them clearly

Check answer

Q3

Send it because the visible prompt is below 8,192 tokensDelete the system message firstReduce the input or output reservation because the combined budget is 8,400 tokensConvert the prompt to fewer characters without counting again

Check answer