Which parts of an LLM application would you implement synchronously, and which would use queues or asynchronous workers? Explain how you would handle backpressure, cancellation, timeouts, retries, ordering, and progress updates for both interactive chat and long-running agent jobs.

How to think about it

I keep bounded validation, retrieval, and one streaming model turn synchronous from the user’s point of view, while slow, fan-out, retryable, or human-dependent work goes through durable queues and asynchronous workers. Interactive chat gets a short deadline and immediate token streaming; long-running agent jobs return a job ID and report durable progress until completion.

Why the boundary exists

“Synchronous” here means the caller is waiting for an answer or a stream. It does not mean the server must block a thread; the service can use asynchronous network I/O while still treating the operation as one request.

A queue separates the producer of work from the consumer. That matters because LLM latency is variable, tool calls can hang, providers impose rate limits, and thousands of tasks may become ready at once. A worker can claim a job, retry it, and resume after a process crash without keeping a browser connection open for an hour.

The queue is not a magic speed button. It adds queueing delay and makes cancellation, ordering, and progress explicit engineering problems. I use it when decoupling is worth those costs.

For interactive chat, the user is already waiting. The critical path should therefore contain only bounded work:

• authenticate and authorize the request;
• load the relevant conversation state;
• retrieve and possibly rerank a bounded number of documents;
• make one model request and stream its output;
• persist the final assistant message.

I might run retrieval and the model call using asynchronous I/O, but I would not hide them behind a general-purpose background queue. The product contract is still “answer this turn now.”

For an agent job, the contract is different. The request validates input, creates a durable job record, and enqueues work. The HTTP response can return 202 Accepted and a job ID immediately. Workers then execute the agent loop—the repeated cycle of model decision, tool call, observation, and next decision—while checkpointing state.

A concrete design

Imagine a contract assistant.

A chat user asks, “Which termination clauses mention a 30-day notice period?” The service loads the last 20 conversation turns, retrieves at most 8 relevant chunks, and starts one streaming model call. Suppose the product budget is 2 seconds to begin the stream and 30 seconds for the complete answer. Those are service-level targets, not promises that every provider call will meet them.

I would limit each tenant to two active generations and keep the model-pool admission queue bounded, perhaps at 20 waiting requests. If that queue is full, the service should reject or shed load with 429 Too Many Requests and a Retry-After value, rather than accept work that will sit unseen for minutes. A smaller answer, cached retrieval, or a cheaper model may be a deliberate overload mode.

The stream itself needs backpressure, meaning the producer slows down or stops when the consumer cannot keep up. A client on a poor connection must not cause the server to buffer unlimited generated tokens. I would use a bounded outbound buffer and, if the model client supports it, pause reading upstream. Otherwise I cancel or abandon the generation after a limit. A disconnected browser should not quietly consume 100,000 tokens.

Now change the request to: “Review 10,000 invoices, extract payment terms, compare them with the vendor contract, and create an exception report.”

That is not a request thread. I would create one job with an idempotency key, split the documents into 100 batches of 100, and enqueue those batch tasks. The key means that repeating the submission does not create a second review job. A worker pool might allow eight active batches for this tenant, subject to the model provider’s rate limit. The final aggregation task runs only after the required batches finish.

The initial request returns quickly. The job may run for 20 minutes or two hours. That is fine because the user is watching a job, not an HTTP socket.

The controls I would make explicit

Backpressure. For chat, enforce per-user and per-tenant concurrency, token limits, request size limits, and a bounded admission queue. Measure queue wait, active generations, stream buffer size, and provider rate-limit responses. For batch jobs, cap queue depth, worker concurrency, and each tenant’s share of capacity. When capacity is exhausted, reject new work or leave it visibly queued. An unbounded queue merely moves the outage from the API to a database table.

Cancellation. Cancellation is cooperative, not telepathic. A chat disconnect should cancel the model stream and any in-flight retrieval when possible. If the provider offers no cancellation, stop consuming the result and mark the generation abandoned; the provider may still charge for work already started.

For a long job, cancellation sets a durable cancel requested state. Workers check it between model calls and tool calls, and before committing side effects. A tool that supports cancellation receives the signal. A tool that does not may finish its current operation, but the worker must avoid starting the next one.

There is a race: cancellation can arrive just after the final commit. The state machine needs terminal states such as completed, failed, and canceled, with a defined rule for that race. “Canceled” should not mean “no external action happened” unless the system can actually guarantee that.

Timeouts. I use an absolute deadline for the whole operation, not a fresh timeout for every retry. If chat has a 30-second deadline and the first model attempt consumes 24 seconds, a retry gets only the remaining budget.

That deadline is divided into smaller budgets: connection timeout, model time-to-first-token timeout, stream idle timeout, retrieval timeout, and tool-call timeout. For a job, I also give each step a timeout and the whole job a maximum runtime or step count. A worker lease—a time-limited claim on a job—must be renewed while the worker is alive; if the worker dies, the job becomes claimable again.

Retries. I retry transient failures such as network interruptions, provider 429 responses, and selected 5xx responses, using exponential backoff with jitter and a maximum attempt count. Jitter prevents every worker from retrying at exactly the same instant.

I do not retry invalid input, authentication failures, deterministic schema errors, or a tool call that may have caused a non-idempotent side effect unless that side effect has an idempotency key or a reconciliation step. Queue delivery is commonly at-least-once, so a worker must assume it can receive the same task twice.

For chat, retrying after partial tokens have reached the user is awkward: a second model response may differ and would duplicate text. I retry before streaming starts where possible. After partial output, I either finish, clearly restart the response, or report failure; I do not silently splice two generations together.

Ordering. I serialize turns within a conversation or assign each turn a sequence number and reject stale writes. Otherwise two simultaneous questions can each read the same old history and produce responses in the wrong order.

Across unrelated conversations, there is no reason to preserve global ordering. For invoice batches, completion order can be arbitrary. The final report should aggregate by document ID, not by the order workers happened to finish. Progress events also carry monotonically increasing sequence numbers so clients can discard duplicates and detect gaps.

Progress. Chat progress is naturally the token stream. I may also show “retrieving documents” before generation, but I avoid pretending that “73 percent of an LLM answer” is meaningful.

For the invoice job, I would persist both current job state and an append-only event history: queued, started, batch completed, tool failed, retry scheduled, and completed. “3,400 of 10,000 invoices processed” is useful because the denominator is known. A model-generated percentage is not. Pub-sub notifications can make the interface feel live, but the durable event log remains the source of truth so a reconnecting client can ask for events after its last sequence number.

The senior-level nuance

Do not put every function call on a queue. A five-millisecond database read and a 40-minute browser automation task have very different operational shapes. Queuing small, bounded steps can add latency, complicate tracing, and make cancellation less reliable.

Conversely, “chat must be synchronous” does not mean every chat turn must finish in one request. If an agent needs a slow web crawl or human approval, stream the immediate plan, persist the conversation, and offer “continue in background.” The user should see the transition instead of watching a spinner until the proxy kills the connection.

A common failure appears first as an old queue age climbing from 10 seconds to 11 minutes, followed by a burst of provider 429 errors. That is usually not solved by adding more retries. It is a backpressure failure or retry storm. Reduce admission, apply per-tenant limits, add jitter, and use a circuit breaker that temporarily stops sending traffic to a failing dependency.

What they’ll ask next

“Can you guarantee exactly-once execution?”
Usually not across a queue and an external API. I design for at-least-once delivery, use idempotency keys, record external operation IDs, and reconcile ambiguous outcomes.

“Why not queue every chat request?”
A small bounded admission queue can protect capacity, but a durable background queue makes latency unpredictable and cancellation harder. Chat should stay on a short streaming path unless the user explicitly accepts background execution.

“How do you make progress trustworthy after a worker crash?”
Checkpoint after meaningful boundaries, persist event sequence numbers, and make retries idempotent. On restart, the worker resumes from the last committed checkpoint rather than trusting an in-memory percentage.

One line to say in the room

“I keep chat bounded and streaming, put slow or fan-out work behind durable queues, and treat backpressure, cancellation, deadlines, idempotency, ordering, and progress as part of the product contract rather than queue configuration details.”