Take one chat request through the autoregressive loop. What is computed once, what is recomputed for each token, how do the KV cache and chat template affect latency and token budget, and why can streaming improve time-to-first-byte without reducing total generation time?

How to think about it

The short answer

Once per request, the server renders and tokenizes the chat template, then runs a prefill forward pass over the entire prompt, producing the initial logits and each transformer layer’s key-value cache. It then generates one token at a time, recomputing the new token’s transformer work and attending to the cached history rather than rerunning the whole prompt; streaming can send the first token as soon as it exists, improving time-to-first-byte without reducing the total generation work.

That is the answer in miniature. The useful detail is understanding exactly what “once” and “each token” mean.

Follow one request through the loop

Suppose a user asks an incident bot:

The checkout API is returning errors. Summarize the likely cause and suggest the first three checks.

The application may have a system message, several earlier turns, and this new user message. A chat template is the model-specific format that turns those messages into one token sequence. It adds role markers, separators, and often a marker saying that the assistant should begin.

The model does not receive a magical array called messages. It receives token IDs.

The serving process then has two phases.

Phase	What happens	Main result
Prefill	The model processes all prompt tokens	Initial logits and KV cache
Decode	The model processes one newly generated token at a time	One next-token distribution per step

Autoregressive means that generation feeds its own previous output back as the next input. The model chooses token one, uses token one to choose token two, and continues until it reaches an end condition such as an end-of-sequence token, a stop sequence, or the output limit.

During prefill, the model processes the whole prompt in one forward pass, conceptually. Prompt tokens can be handled in parallel on a GPU, subject to attention and memory limits. The model still applies every transformer layer to every prompt position. It does not skip the work; it simply has all the input positions available at once.

The final prompt position produces logits, which are the model’s unnormalized scores for every token in its vocabulary. The server turns those scores into the first generated token using its decoding rule, such as greedy selection, temperature sampling, or top-p sampling.

For a prompt of length P and an answer of N tokens, the simple picture is:

1. Run prefill over P prompt tokens.
2. Use the final prompt logits to select generated token one.
3. Run one-token decode to select generated token two.
4. Repeat until the answer stops.

So an answer of 200 tokens normally needs one prompt pass plus about 199 one-token decode passes. Details vary with batching and implementation, but the first generated token can come directly from the prefill logits. There is no need to run a separate decode pass just to rediscover it.

The model weights are not loaded or recalculated for each token. They are normally loaded into the serving process once and reused by every request. “Computed once” here means the prompt-side work for this request, not rebuilding the model.

What the KV cache actually saves

At each transformer layer, self-attention creates:

• a query, which asks what the current position should look at;
• a key, which describes what a position offers for matching;
• a value, which contains the information retrieved when it is attended to.

The KV cache stores the key and value vectors for tokens already processed in this request.

When the first generated token is processed, the model computes its new query, key, and value at every layer. Its query attends to the keys and values for the entire prompt. The new key and value are appended to the cache.

For the next token, the model computes new activations again through every layer. It creates a new query, key, and value, then attends to all cached keys and values, including the previous generated token. The old keys and values are reused. Their projections do not have to be calculated again.

The query for an old token is not cached because future steps do not need to ask that old token a new question. Future steps need the old keys and values.

Warning: The KV cache does not make attention free, and it is not long-term memory. Every new token still attends to the cached sequence, so a longer context makes each decode step more expensive. The cache removes repeated key and value computation; it does not remove the need to read the history.

There is a real memory trade-off. A rough cache-size formula is:

2 × layers × KV heads × head dimension × bytes per value × sequence length

The first factor of two is for keys and values.

Imagine a model with 32 layers, 8 KV heads, head dimension 128, and an FP16 cache. At 1,200 cached positions:

2 × 32 × 8 × 128 × 2 × 1,200 = 157,286,400 bytes

That is roughly 150 MiB for one sequence, before allocator overhead. If 32 users are generating at once, that part alone is roughly 4.7 GiB. The cache grows as the answer grows.

Grouped-query attention and multi-query attention reduce cache size by using fewer key-value heads than query heads. Cache quantization can reduce it further, with implementation and quality trade-offs. This is why a model that fits in GPU memory for one request may struggle badly under concurrent traffic.

Without a KV cache, a naive decoder would repeatedly process the entire prompt-plus-answer prefix at every step. The symptom is familiar: the answer begins at a reasonable speed, then each additional token takes longer. With a cache, decoding is still slower for long contexts because each new query scans more history, but the redundant K/V projections are avoided.

Chat templates change both latency and budget

Suppose the content of our incident-bot conversation tokenizes to 1,000 tokens. Its chat template adds 38 tokens for system and user-role markers, separators, and the assistant-generation marker. The actual model input is therefore 1,038 tokens, not 1,000.

If the application allows up to 200 new tokens, the request may use as many as 1,238 positions in the model’s context. With a 4,096-token context limit, that fits. If conversation history grows until the rendered prompt is 3,950 tokens, a 200-token answer no longer fits. The serving stack may reject the request, truncate history, or reduce the available output, depending on its policy.

A token budget can mean two related limits:

• input tokens: the rendered prompt, including template control tokens;
• output tokens: the maximum number of new tokens the model may generate.

The context window must accommodate both. Providers may also bill or meter input and output separately, but the exact accounting is provider-specific.

Templates are not cosmetic. The model was trained on a particular conversation format. A missing assistant-generation marker can cause the model to continue the user text, emit role markers, or produce strangely formatted output. A verbose system prompt increases prefill work, consumes context, and gives every generated token a longer cache to attend over.

The exact token count must come from the model’s tokenizer after rendering the template. Counting characters, words, or messages is not reliable.

Why streaming improves first-byte latency

Assume prefill takes 0.8 seconds and the decoder produces 25 tokens per second. A 180-token answer takes about 7.2 seconds of decoding, so the complete response arrives after roughly 8 seconds, ignoring network details.

With a non-streaming response, the server commonly buffers those 180 tokens and sends the body only after generation finishes. The client sees its first useful byte near the 8-second mark.

With streaming, the server can flush the first available token or chunk after prefill and then send more chunks as decoding continues. The client might see the first token at about 0.84 seconds, while the final token still arrives around 8 seconds. The model performed essentially the same prefill and decode work. The bytes were merely delivered earlier instead of held behind the finish line.

Strictly speaking, time-to-first-byte can include HTTP headers sent before inference begins. Time-to-first-token is often the more meaningful model metric. Streaming may send headers immediately, but it cannot normally display meaningful generated text until prefill has produced the first-token logits.

Streaming can even add a small amount of overhead from flushing, serialization, and client-side rendering. Its main benefit is lower waiting time for the user, not fewer FLOPs. It also lets a client cancel a response after seeing enough text, which can reduce work in practice, but that is a cancellation decision, not a property of streaming itself.

The senior-level nuance

The textbook split is “prefill once, decode one token at a time.” Real servers complicate it with continuous batching, where requests at different stages share GPU work, and prefix caching, where an identical prompt prefix may reuse KV states across requests.

Prefix caching changes what is computed once across requests, but only when the prefix is genuinely identical in the way the serving system requires. A changed system prompt, different template version, or altered tokenization can invalidate the reuse. Cache entries also need correct sequence and positional handling; casually reusing a cache across unrelated users is both a correctness bug and a data-isolation incident.

The main latency levers therefore differ:

• Long prompts mainly hurt time to first token because prefill must process them.
• Long contexts also slow every later token because each query reads more cached history.
• Long answers cost serial decode time.
• Large concurrency turns KV-cache memory into a capacity constraint.
• Streaming improves when the user sees output, not how quickly the model finishes.

What they’ll ask next

Why not cache the queries too?
Because future tokens need to compare their new query with old keys and retrieve old values. They do not need to reuse old queries.

Does the KV cache make generation constant-time per token?
No. It removes repeated prefix computation, but each new token still attends over the growing cache and passes through every transformer layer.

What would you optimize if the first token is slow but later tokens are acceptable?
Inspect prompt length, chat-template overhead, queueing, and prefill efficiency. If the first token is fast but later tokens slow down, inspect context length, KV-cache memory pressure, decode throughput, and concurrency.

One line to say in the room

“I’d separate prefill from decode: prefill processes the rendered prompt once and builds the KV cache, decode advances one token at a time using that cache, while the template changes the real token budget and streaming changes when bytes arrive, not how much generation work is required.”