Async patterns for LLM apps

An LLM call is slow I/O — anywhere from 500 milliseconds to 30 seconds, dominated by the network and the model’s own inference time. The happy consequence is that fanning out ten calls in parallel costs roughly the same wall-clock as making one. If you are building an LLM service and not using asyncio, you are leaving the single biggest latency win on the table.

Why asyncio and not threads? Threads give parallelism, but each one ties up an OS thread while it waits, and the GIL serialises any CPU work anyway. asyncio runs on a single thread: while one coroutine awaits a network response, the loop runs others. For I/O-bound work — which an LLM call always is — you can keep hundreds of calls in flight with no thread overhead. The examples below use small mock async calls so they run anywhere and produce exact output; in production you swap them for openai.AsyncOpenAI, anthropic.AsyncAnthropic, or whatever async client your provider ships — the patterns are identical.

The shape of every LLM call

async def complete(client, prompt):
    resp = await client.messages.create(
        model="claude-opus-4",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=512,
    )
    return resp.content[0].text

It is a single await — that is the entire async surface, and it sits exactly where you want it, on the slow part.

Pattern 1 — fan-out for ensembles

A reliable quality trick is to ask the model the same question several times (with a temperature above 0) and aggregate, or to ask several different models at once. Either way you want all N requests in flight together — so gather them:

import asyncio
from collections import Counter

# A mock returning a FIXED label per call, so the demo is exact.
# In production: openai.AsyncOpenAI / anthropic.AsyncAnthropic.
async def mock_complete(prompt, call_id):
    await asyncio.sleep(0.05)
    votes = ["positive", "positive", "negative", "positive", "neutral"]
    return votes[call_id]

async def ensemble_classify(prompt, n=5):
    results = await asyncio.gather(
        *[mock_complete(prompt, i) for i in range(n)]
    )
    return Counter(results).most_common(1)[0][0], results

label, votes = asyncio.run(ensemble_classify("Was the review good?"))
print("votes:", votes)
print("label:", label)

votes: ['positive', 'positive', 'negative', 'positive', 'neutral']
label: positive

The five calls each “wait” 0.05s, but because gather overlaps them, the wall-clock is about one call’s worth, not five. That is what makes ensembling cheap in latency terms even though it is expensive in tokens — and a 3-to-1 majority vote is a simple, effective reliability win.

Pattern 2 — streaming with async generators

LLMs emit tokens one at a time, and streaming lets you render the response as it arrives instead of waiting for the whole thing. In async Python the shape is an async generator you iterate with async for:

import asyncio

async def mock_stream(prompt):
    chunks = "The capital of France is Paris .".split()
    for chunk in chunks:
        await asyncio.sleep(0.02)
        yield chunk + " "

async def main():
    full = []
    async for token in mock_stream("Capital of France?"):
        print(token, end="", flush=True)
        full.append(token)
    print()
    print("complete:", "".join(full))

asyncio.run(main())

The capital of France is Paris . 
complete: The capital of France is Paris . 

Two things make it work: async def combined with yield produces an async generator, and the consumer drives it with async for token in stream:, where each iteration is itself an await. Production code does exactly what the example does — it both yields each token to the UI and accumulates the full response to store.

Pattern 3 — Semaphore for rate limits

Every provider rate-limits by requests-per-minute and tokens-per-minute, so naively gather-ing a thousand prompts will earn you a 429 within seconds. The fix is to cap concurrency with asyncio.Semaphore: schedule everything, but let only N tasks make their actual call at once.

import asyncio

MAX_CONCURRENT = 4

async def mock_complete(prompt):
    await asyncio.sleep(0.05)
    return f"answer to: {prompt}"

async def bounded(sem, prompt):
    async with sem:                # blocks if 4 are already in flight
        return await mock_complete(prompt)

async def main():
    prompts = [f"Q-{i}" for i in range(20)]
    sem = asyncio.Semaphore(MAX_CONCURRENT)
    results = await asyncio.gather(*[bounded(sem, p) for p in prompts])
    print("count:", len(results))
    print("first:", results[0])
    print("last: ", results[-1])

asyncio.run(main())

count: 20
first: answer to: Q-0
last:  answer to: Q-19

All twenty are scheduled at once, but the async with sem: gate admits only four at a time — so the batch runs as five waves of four (about 5 × 0.05 = 0.25s) instead of a single thundering stampede. You get smooth, bounded parallelism with no manual batching.

Pattern 4 — bounded queues for back-pressure

When a producer (a Kafka consumer, a file reader, a crawler) feeds work to LLM workers, you want the producer to stop reading when the workers fall behind — otherwise an unbounded queue quietly fills RAM until the process is killed. asyncio.Queue(maxsize=N) blocks put when full, pausing the producer automatically:

import asyncio

async def mock_complete(prompt):
    await asyncio.sleep(0.02)
    return prompt.upper()

async def producer(q, n):
    for i in range(n):
        await q.put(f"prompt-{i}")   # blocks here when the queue is full
    for _ in range(3):
        await q.put(None)            # one sentinel per worker, to stop them

async def worker(q, out):
    while True:
        item = await q.get()
        if item is None:
            return
        out.append(await mock_complete(item))

async def main():
    q = asyncio.Queue(maxsize=5)     # back-pressure window of 5
    out = []
    async with asyncio.TaskGroup() as tg:
        tg.create_task(producer(q, 15))
        for _ in range(3):
            tg.create_task(worker(q, out))
    print("processed:", len(out))
    print("sample:", sorted(out)[:3])

asyncio.run(main())

processed: 15
sample: ['PROMPT-0', 'PROMPT-1', 'PROMPT-10']

The producer queues fifteen prompts but never gets more than five ahead of the workers, because q.put blocks on a full queue. That single maxsize is the cheapest back-pressure mechanism in async Python, and it is how high-throughput LLM services stay memory-stable. (We sorted the sample only because three workers finish in a scheduling-dependent order.)

Pattern 5 — timeouts that cancel

LLM endpoints occasionally stall, and one stuck request that holds a semaphore slot forever will quietly collapse your throughput. You need a hard per-call ceiling, and asyncio.wait_for provides it by cancelling the inner coroutine when the deadline passes:

import asyncio

async def slow_completion():
    await asyncio.sleep(5)           # pretend the model hung
    return "never seen"

async def main():
    try:
        result = await asyncio.wait_for(slow_completion(), timeout=0.3)
        print(result)
    except asyncio.TimeoutError:
        print("timed out — moving on")

asyncio.run(main())

timed out — moving on

wait_for(coro, timeout) cancels coro if it overruns, which raises asyncio.CancelledError inside it — so a well-behaved client uses try/finally to release resources (close sockets, return semaphore permits) on cancellation.

Putting it together — a production worker

A real “call the LLM for a batch of inputs” worker combines all of the above: a semaphore for concurrency, wait_for for a per-call timeout, graceful failure, and a TaskGroup for a clean exit. Here two specific calls are rigged to hang, so the output is exact:

import asyncio

async def mock_complete(prompt, hang):
    await asyncio.sleep(10 if hang else 0.05)
    return f"answer to {prompt}"

async def call_one(sem, prompt, hang, timeout):
    async with sem:
        try:
            return await asyncio.wait_for(mock_complete(prompt, hang), timeout)
        except asyncio.TimeoutError:
            return f"TIMEOUT {prompt}"

async def process_batch(prompts, hang_ids, max_concurrent=4, per_call_timeout=0.5):
    sem = asyncio.Semaphore(max_concurrent)
    async with asyncio.TaskGroup() as tg:
        tasks = [
            tg.create_task(call_one(sem, p, i in hang_ids, per_call_timeout))
            for i, p in enumerate(prompts)
        ]
    return [t.result() for t in tasks]

prompts = [f"Q-{i}" for i in range(12)]
results = asyncio.run(process_batch(prompts, hang_ids={3, 7}))
for r in results:
    print(r)

answer to Q-0
answer to Q-1
answer to Q-2
TIMEOUT Q-3
answer to Q-4
answer to Q-5
answer to Q-6
TIMEOUT Q-7
answer to Q-8
answer to Q-9
answer to Q-10
answer to Q-11

The two hanging calls (Q-3, Q-7) hit the 0.5s ceiling and returned TIMEOUT rather than stalling the whole batch; everything else completed. Swap mock_complete for the real provider’s async method and that is a production-ready worker — semaphore for concurrency, wait_for for the per-call timeout, TaskGroup for a clean exit on failure.

In one breath

• Every LLM call is one slow await — asyncio keeps many in flight on one thread.
• gather fans out parallel completions; an async generator with async for streams tokens.
• asyncio.Semaphore(N) caps concurrency to respect rate limits; a bounded asyncio.Queue gives back-pressure.
• asyncio.wait_for(coro, timeout) cancels a stalled call — always set per-call timeouts.
• The production worker = Semaphore + wait_for + TaskGroup.

Practice

What’s next

Most production LLM services use asyncio for the I/O layer and reach for a process pool only for the occasional CPU-bound preprocessing step. Threads sit in the middle — mostly used through asyncio.to_thread to wrap stubborn synchronous libraries with no async API. Next we turn from how the code runs to the data it validates: Pydantic.