Design an AI gateway that fronts several model providers. How would it handle authentication, policy enforcement, routing, retries, provider outages, circuit breaking, fallback models, streaming failures, and the risk that retries multiply cost or duplicate tool actions?

How to think about it

Put a single gateway in front of every model provider, with provider adapters behind it, and make it the enforcement point for identity, policy, routing, budgets, and telemetry. Give each request a deadline and idempotency identity; retry only failures that are safe to retry, use per-provider circuit breakers and capability-aware fallbacks, and never pretend a streamed or side-effecting request can be transparently replayed.

The shape of the system

The request path is:

client -> gateway -> policy -> router -> provider adapter -> model provider

The gateway exposes one internal contract to callers. An adapter translates that contract into each provider’s request and response format. That translation layer is important: it keeps provider-specific authentication, error codes, streaming formats, and tool-call details out of every application.

The gateway should be stateless on the hot path, but it still needs durable supporting stores:

• configuration and secrets, normally backed by a secret manager;
• rate-limit and budget counters;
• an idempotency or operation ledger;
• circuit-breaker state, shared carefully across gateway instances;
• metrics, traces, and redacted audit records.

“Stateless” does not mean “forgetful.” A retrying system that forgets what it already did is how a refund happens twice.

Authentication comes before model access

The client authenticates to the gateway, not directly to a model provider. Depending on the caller, that might be an OAuth access token, a service API key, or mutual TLS between trusted services.

For a JWT, the gateway validates the signature, issuer, audience, expiry, and key identifier. It maps the authenticated subject to a tenant, application, and permissions. Authentication answers “who are you?” Authorization answers “may you use this model, region, token budget, or tool?”

Provider credentials stay server-side in the secret manager. The gateway selects the appropriate provider key and injects it into the adapter. It must not forward a user’s raw provider key, and it should not put provider secrets in logs, traces, error messages, or client-visible metadata.

Every request gets a gateway request ID. That ID is useful for tracing, but it is not automatically an idempotency key. The gateway should accept a caller-supplied operation key for requests whose effects may be repeated, or create one and persist it when the gateway owns the workflow.

Policy is more than prompt moderation

Policy is enforced before the request spends money or reaches a provider. Typical decisions include:

• Is this tenant allowed to use this model and region?
• Is the context within the model’s limit?
• Is the requested output budget within the tenant’s quota?
• May this request contain personal or regulated data?
• Which tools may this user invoke, with which arguments?
• Should prompts, outputs, and tool payloads be logged, redacted, or not retained?
• Does the response need output safety checks before delivery?

Run cheap, deterministic checks first: authentication, authorization, size limits, quotas, and model capability. Run expensive content or data-loss checks only where the policy requires them.

Apply policy to tool arguments as well as natural-language text. A model can produce a perfectly polite tool call that attempts to transfer $50,000. Validate the operation, target, amount, and user permission in the tool service. The model is not an authorization boundary.

Route by capability, then by health

Routing starts with hard constraints, not price. A request may require a long context window, image input, structured output, tool calling, a particular data region, or a contractual provider. Eliminate providers that cannot satisfy those requirements.

Among the survivors, score live signals such as recent error rate, latency, capacity, and cost. Keep those signals separate by provider, model, and region. A provider can be healthy for short text generation and failing for long-context requests.

Do not normalize away every difference. A common request shape is useful, but the gateway should preserve provider-specific finish reasons, usage uncertainty, safety metadata, and request IDs for operators. “Portable” does not mean “identical.” Two models given the same prompt may produce different tool arguments, refusal behavior, or output quality.

A total request deadline travels through the system. If the caller allows eight seconds, the gateway cannot spend seven seconds retrying and then ask a provider for another eight. Each attempt receives the remaining deadline.

Retry narrowly, because a timeout is ambiguous

A retry is reasonable when the failure probably happened before the provider accepted the request, or when the provider explicitly indicates a transient failure. Examples include a connection failure before sending the body, a temporary server error, or a rate-limit response that includes a retry delay.

Do not retry authentication failures, invalid requests, policy denials, unsupported capabilities, or a tenant whose quota is exhausted. Those failures will not improve when repeated.

The dangerous case is a timeout after the request was sent. The gateway cannot tell whether the provider rejected it, completed it, or completed it and lost only the response. Retrying may create a second billable generation. A gateway-generated request ID does not solve this unless the provider itself supports idempotency for that operation.

Use a small attempt limit, exponential backoff with jitter, and a total deadline. Jitter matters because a fleet of gateways that all waits exactly one second will hit an already struggling provider together. Respect a provider’s Retry-After signal when it is present, but never let it override the caller’s deadline or the gateway’s retry budget.

A useful decision table looks like this:

Failure	Default action
Invalid credentials or request	Return the error; do not retry
Connection failed before send	Retry once if the deadline allows
Rate limit	Wait or route elsewhere, within budget
Server error or timeout before headers	Retry only under a bounded policy
Timeout after send	Retry only with a safe idempotency design
Stream failed after bytes arrived	Do not silently switch providers

Outages, circuit breakers, and fallback

Use passive health signals from real traffic and occasional bounded probes. Track failures separately for each provider-model-region combination. A circuit breaker has three useful states:

• Closed: traffic flows normally.
• Open: requests fail fast or use a fallback; calls are not sent to the failing provider.
• Half-open: after a cool-down, a small number of probes test recovery.

Open the circuit based on a meaningful error window, not one unlucky timeout. Exclude client errors from outage detection. A tenant-specific 429 should not take down the provider for everyone. Conversely, a rising rate of timeouts and server errors across many tenants is strong evidence of a provider problem.

Pair breakers with bulkheads: separate concurrency limits and queues by provider or model family. Otherwise a slow provider can consume every gateway worker before the breaker notices.

Fallback is a semantic decision. Choose a model with the required context, modality, tool support, region, and acceptable quality. A text-only model is not a valid fallback for a vision request. If no equivalent model exists, a clear degraded response is safer than silently removing a required capability.

Tell callers when fallback occurred, at least through response metadata and telemetry. The answer may be valid while its quality, latency, or data-processing location differs from the primary path.

A concrete failure at 3 a.m.

Imagine a checkout assistant with an eight-second deadline. The request contains 4,200 input tokens, permits 700 output tokens, and asks the model to use a book_flight tool. Provider A is preferred; Provider B supports the same tool contract; Provider C is cheaper but cannot call tools.

The gateway authenticates the application, checks that the tenant may book flights, confirms that Provider A and B meet the region and tool requirements, and assigns an operation key such as booking-8f31. It sends the request to A.

A returns a rate-limit response with a 1.5-second retry delay. The router may wait if enough deadline remains, or send the request to B if B is healthy and the tenant’s policy allows it. It does not send the request to C merely because C is available.

Now consider the harder failure. A accepts the request, generates a tool call, and the connection dies before the gateway receives the final response. Retrying the model request can produce the same book_flight intent again. Retrying the tool call can buy two tickets.

The tool executor therefore needs its own durable idempotency record. The stable operation key identifies the booking step, not merely the model’s latest tool-call ID. If the same operation arrives again, the executor returns the recorded result instead of performing a second booking. The record must be written around the side effect with a design appropriate to the tool, often using a database uniqueness constraint or an outbox-style workflow.

Exactly-once execution across independent networks is not something a gateway can magically promise. At-most-once execution can lose an action; at-least-once execution can duplicate it. Idempotent effects and a durable operation ledger make retries safe enough for practical systems.

Streaming changes the fallback rule

Before the first response bytes arrive, the gateway can retry or fail over, subject to the same ambiguity and deadline rules. After it has sent tokens to the client, it cannot switch to another model invisibly. The client has already seen a prefix, and the second model will not reliably continue with the same token sequence.

The gateway should propagate cancellation when the client disconnects, enforce an idle timeout, apply backpressure, and record whether the stream ended normally or with incomplete usage information. On a mid-stream failure, it should send a clear stream error if the protocol permits one and let the client decide whether to start a continuation request.

Never execute a streamed tool call merely because its first argument fragment appeared. Buffer and validate the complete structured arguments, then execute through the idempotent tool layer.

What they’ll ask next

How do you avoid a retry storm?
Use exponential backoff with jitter, bounded attempts, concurrency limits, deadlines, and circuit breakers. Retries consume capacity, so they need their own rate and budget controls.

How do you distinguish an outage from a bad prompt?
Classify errors by status and provider metadata, then compare them across tenants, models, and regions. Invalid-request failures are local; correlated timeouts and server errors are outage signals.

Can the gateway guarantee exactly-once tool execution?
No, not across a network boundary. It can provide durable idempotency keys, deduplication, and a tool workflow that makes repeated delivery safe.

One line to say in the room

“I’d make the gateway the policy and reliability boundary, but I would never hide uncertainty: retries are bounded, fallbacks are capability-aware, streams fail honestly, and every side effect is idempotent.”