The autoregressive loop

When you call an LLM, it doesn’t think for a moment and then emit a paragraph. It produces one token, then another, then another — each one a function of every token before it. That’s the whole generation process. Everything about latency, streaming, and cost falls out of this single fact.

Because each token is predicted from all prior tokens, the model can’t generate token 5 until token 4 exists. Output is inherently sequential — no amount of hardware parallelism can eliminate that dependency.

The loop, in plain Python

Here is the algorithm, with no neural network in sight — a tiny transition table stands in for the model’s probabilities, and we pick greedily (always the most likely token):

# A real LLM outputs a probability distribution over ~100k tokens.
# We fake one with a tiny transition table for "The capital of France is __".
def mock_next_token_probs(tokens_so_far):
    last = tokens_so_far[-1] if tokens_so_far else ""
    table = {
        "is":    {"Paris": 0.92, "located": 0.05, "the": 0.03},
        "Paris": {".": 0.85, ",": 0.10, " a": 0.05},
        ".":     {"<END>": 0.99, "\n": 0.01},
    }
    return table.get(last, {"<END>": 1.0})

def generate(prompt_tokens, max_tokens=20, stop=("<END>",)):
    tokens = list(prompt_tokens)
    for step in range(max_tokens):
        probs = mock_next_token_probs(tokens)
        next_tok = max(probs, key=probs.get)   # greedy: the highest-probability token
        if next_tok in stop:
            break
        tokens.append(next_tok)
        print(f"step {step}: emitted {next_tok!r}")
    return tokens

result = generate(["The", "capital", "of", "France", "is"])
print("\nfinal:", " ".join(result))

step 0: emitted 'Paris'
step 1: emitted '.'

final: The capital of France is Paris .

The loop produced “Paris”, then ”.”, then hit the <END> token and stopped — exactly the three-step sequence the diagram showed. Every commercial LLM is doing precisely this — just with a neural network producing the probabilities, and a vocabulary of ~100,000 tokens instead of three.

Why this matters: streaming is free

Because tokens come out one at a time, the API can stream them to you the moment each is produced. That’s why ChatGPT, Claude, and Gemini all show the response appearing word-by-word — it’s not a UX gimmick, it’s the natural shape of the underlying process.

Stop conditions

Generation ends one of three ways:

1. End-of-text token — the model emits its special end token (e.g. <|endoftext|>). This is the “natural” stop.
2. max_tokens hit — you set a hard ceiling. Common default: 1024 or 4096. Always set this; otherwise a runaway loop can cost dollars.
3. Stop sequence matched — you pass stop=["\nUser:", "###"] and the model emits one of those strings. Useful for few-shot prompts where you want to prevent the model from continuing the pattern.

# A stop-sequence check during streaming.
stream = ["Sure", "!", " Here", " is", " the", " answer", ":", " 42",
          "\n", "User", ":", " next", " question"]

stop_sequences = ["\nUser:"]
buffer = ""
for tok in stream:
    buffer += tok
    if any(s in buffer for s in stop_sequences):
        for s in stop_sequences:           # trim the stop sequence and end
            if s in buffer:
                buffer = buffer.split(s)[0]
        print("hit stop sequence")
        break
    print(f"emit: {tok!r}")

print("\nfinal output:", repr(buffer))

emit: 'Sure'
emit: '!'
emit: ' Here'
emit: ' is'
emit: ' the'
emit: ' answer'
emit: ':'
emit: ' 42'
emit: '\n'
emit: 'User'
hit stop sequence

final output: 'Sure! Here is the answer: 42'

The stop sequence \nUser: only completed once both the \n, User, and : tokens had arrived — so the model emitted right up to the boundary, then the buffer was trimmed back to the clean answer.

Implications for your code

A few things follow directly from “one token at a time”:

• Cost = output tokens × output price + input tokens × input price. Output tokens are usually 3-5x more expensive than input tokens because they require sequential generation.
• Total latency ≈ TTFT + (output_tokens / tokens_per_second). On GPT-4-class models you’ll see roughly 50-150 tokens/sec depending on the model. A 500-token response takes 3-10 seconds.
• Long outputs are slow, regardless of model size. Asking for a 10,000-token summary doesn’t get faster on a bigger GPU — it’s inherently sequential. If you need speed, ask for less.
• The model can’t “go back”. Once a token is emitted, that’s it. This is why prompt engineering matters: a bad first token cascades.

What this is not

This is not “the model thinks then writes”. There’s no internal draft the model is transcribing. Each token is a fresh forward pass through the network. The “reasoning” you see in chain-of-thought outputs is real reasoning — but it’s happening as the tokens are emitted, not before. This is why asking for reasoning before the answer actually improves correctness: the model gets to use those tokens as scratch space.

In one breath

• Generation is a loop: predict the next token from all prior tokens, sample it, append it, repeat.
• Output is inherently sequential — no GPU parallelism removes the token-by-token dependency.
• It stops three ways: an end-of-text token, max_tokens, or a matched stop sequence.
• Streaming is free (it just exposes the loop) and time-to-first-token is what users feel.
• Put reasoning before the answer — the model uses those tokens as scratch space; a prefilled { steers it into JSON.

Quick check

Next

The next lesson covers sampling — how the model picks which token from its probability distribution. That’s where temperature, top-p, and the difference between deterministic and creative outputs come from.