Streaming LLM responses

A non-streaming LLM call is brutal UX. The user clicks send, stares at a spinner for four seconds, and then a two-hundred-token answer materialises all at once. With streaming, the first words appear in roughly 300 milliseconds and the rest flows in after. The total time is identical — but the feel is completely different, and that difference is why every modern chat product streams by default.

Time-to-first-token is the real metric

For an interactive product, TTFT — time to first token — matters more than total generation time. A five-second response that starts in 300ms feels faster than a two-second response that starts in 1.5s, because your brain registers “the system is responding” as the win, not “the system finished”. The picture makes the asymmetry plain — same length, very different first moment:

This is not a UX nicety — it is why ChatGPT, Claude.ai, and every serious chat product stream. If your app is interactive and you are not streaming, you are shipping the worst version of the product.

The mechanism — server-sent events

When you stream, the SDK opens a long-lived HTTP connection and the server pushes server-sent events (SSE) — a standard HTTP protocol where the server emits newline-delimited data: frames over a single persistent response — as the model generates. Each event is a small JSON delta, and the connection stays open until the model finishes or you close it. On the wire it looks like this:

event: response.output_text.delta
data: {"delta": "Hello"}

event: response.output_text.delta
data: {"delta": " world"}

event: response.completed
data: {"response": ...}

You never parse this by hand — the SDK hands you an iterator — but knowing it is just SSE explains everything downstream: why streams work through any HTTP proxy, why they buffer poorly on some CDNs, and why a disconnect looks like an ordinary dropped connection.

Consuming a stream

The two SDKs follow the same shape — a context-managed stream, an iterator of deltas, and a final aggregated object once it completes:

# OpenAI
with client.responses.stream(
    model="gpt-5-mini",
    input=[{"role": "user", "content": "Write a haiku about Python."}],
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
    final = stream.get_final_response()

# Anthropic
with client.messages.stream(
    model="claude-sonnet-4.6",
    max_tokens=512,
    messages=[{"role": "user", "content": "Write a haiku about Python."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()

To see the consume-and-aggregate pattern resolve without a live API, here is a generator that mirrors the SDK’s iterator — also exactly how you would build a fake LLM client for local tests:

# A stand-in for the SDK's stream of text deltas.
def mock_stream(text, chunk_size=6):
    for i in range(0, len(text), chunk_size):
        yield text[i:i + chunk_size]

full_response = "Streaming makes the wait feel instant — the user sees words appear, not a spinner."

# Flush each delta to the UI AND accumulate the full string, in one pass.
aggregated = []
print("Streaming output:")
for delta in mock_stream(full_response):
    print(delta, end="", flush=True)   # straight to the UI
    aggregated.append(delta)           # also build the final string

print()
print()
print(f"Final aggregated length: {len(''.join(aggregated))} chars")
print(f"Final response: {''.join(aggregated)!r}")

Streaming output:
Streaming makes the wait feel instant — the user sees words appear, not a spinner.

Final aggregated length: 82 chars
Final response: 'Streaming makes the wait feel instant — the user sees words appear, not a spinner.'

The one rule that matters: flush each delta to the UI as it arrives while also appending it to an aggregator you will persist or log when the stream ends. Do it in a single pass — split it into two and you throw away the latency win you came for.

Aggregating structured streams

Plain text deltas are easy. Streaming a structured output (JSON, tool calls) is harder, because the partial JSON is not valid until the final brace closes. The pattern, when you need it, is: accumulate the raw text, try to parse it incrementally with a tolerant parser (json5, partial-json, or your own), and update the UI optimistically with whatever fields have stabilised. But most apps do not need this.

Cancellation — the part most code skips

If the user navigates away mid-stream, you should close the connection and stop generating. Real cancellation needs three things working together: a signal from the UI (a websocket close, an AbortController), a way to break out of the iterator without losing the aggregated state, and a clean teardown of the SDK’s HTTP connection. Here the cancel fires after a fixed number of chunks so the demo is reproducible; in production a UI thread flips the flag asynchronously:

class StreamCancelled(Exception):
    pass

def mock_stream(text, chunk_size=4):
    for i in range(0, len(text), chunk_size):
        yield text[i:i + chunk_size]

def run_stream(text, cancel_after):
    aggregated = []
    try:
        for n, delta in enumerate(mock_stream(text)):
            if n == cancel_after:          # the UI sets this asynchronously in real life
                raise StreamCancelled()
            print(delta, end="", flush=True)
            aggregated.append(delta)
    except StreamCancelled:
        print("\n[stream cancelled by user]")
    finally:
        pass                               # in production: stream.close()
    return "".join(aggregated)

text = "The fox jumps over the lazy dog and keeps on going for a very long time indeed."
partial = run_stream(text, cancel_after=7)   # user leaves after 7 chunks
print(f"Aggregated before cancel: {partial!r}")

The fox jumps over the lazy 
[stream cancelled by user]
Aggregated before cancel: 'The fox jumps over the lazy '

The structure is the whole point: a flag checked every iteration, and a finally block that always releases the connection. In production the flag check becomes AbortController.signal.aborted on the web, or a websocket-disconnect handler server-to-server — but the shape is identical to this.

Async, when you need throughput

If you are fanning out many streams at once — a server handling fifty concurrent chats — use AsyncOpenAI / AsyncAnthropic with asyncio: the iterator becomes an async for, and you can run many streams concurrently with asyncio.gather. Same patterns, just non-blocking. For a single user-facing endpoint the sync API plus a thread pool is usually enough; reach for async when you are proxying hundreds of streams through one process.

In one breath

• Streaming cuts time-to-first-token, not total time — and TTFT is what users feel.
• Under the hood it is server-sent events: newline-delimited data: deltas over one long HTTP response.
• Consume the iterator and flush-while-aggregating in one pass; grab the final object when it completes.
• Stream prose; batch structured JSON (partial JSON is not worth parsing).
• Support cancellation with a per-iteration flag and a finally that closes the stream — you are still billed for delivered tokens.

Practice

What’s next

You can call models and stream their answers. Last in this track: turning text into vectors with embeddings — the foundation of search, recommendations, and retrieval-augmented generation.