AI gateways

At 3:07 a.m., your support assistant starts returning errors. The primary model provider is having an outage. Your billing service still works. Your database still works. The assistant is the only thing on fire, but six application teams own six slightly different copies of its model-calling code.

One service retries three times. Another retries forever. A third has an old API key. None of them knows how much of the month’s model budget remains. Finance will find out eventually, probably from the credit-card notification.

An AI gateway puts one proxy, a service that receives a request and forwards it elsewhere, in front of those model calls. Applications call the gateway. The gateway chooses a provider, applies common policies, translates request formats, records what happened, and returns the result.

It is the API-gateway pattern with token counts, streaming, model fallbacks, and a much more interesting bill.

One address, many backends

Imagine a company called Acme. Its support assistant is used by the web app, the mobile app, and an internal agent tool. Product wants the cheapest suitable model for routine questions. Legal wants prompts and outputs handled under one retention policy. Finance wants a daily budget for each team. Reliability wants a fallback when the main provider returns an HTTP 503, which means “service unavailable.”

Without a gateway, each application must implement all of that. With a gateway, each application sends a request to one internal endpoint, such as llm.acme.internal. The gateway may forward that request to a hosted provider, another hosted provider, or a self-hosted inference server.

A gateway usually exposes an OpenAI-compatible API, meaning it accepts a widely used request and response shape so client code can use one interface. That does not mean providers are identical. Tool calling, reasoning controls, multimodal inputs, safety settings, streaming events, and usage fields still differ. The gateway’s provider adapter translates between those dialects and must expose differences that affect behavior.

The gateway is not merely a load balancer. A load balancer spreads traffic among equivalent servers. A model gateway may choose a different model because the request is simple, the customer has a restriction, the primary provider is unhealthy, or the team has spent its budget.

What happens to one request

Use Acme’s “Where is my replacement order?” question. A production gateway commonly handles it like this:

1. Authenticate and identify the caller. Check the application’s credential and map it to a team, project, user, or service account. Provider keys stay in the gateway.

2. Normalize the request. Validate its shape, apply a model alias such as support-default, and convert it into the provider-specific format. Applications can keep using the alias while operators change its backend by configuration.

3. Apply policy. Check request and token limits, allowed models, data-handling rules, and remaining budget. Reject before spending money when necessary.

4. Route the request. A classifier, application hint, tenant rule, health signal, or cost policy selects a backend. A routine order-status question might use a cheaper model; a complicated contract question might use a stronger one.

5. Execute within a deadline. Set connection and total-request timeouts. Retry a safe transient failure or try a configured fallback.

6. Normalize and record. Return the gateway’s public response format and record latency, status, selected model, provider, token usage, and cost attribution.

The exact order can vary—for example, a cache may be checked before routing—but the decisions should be consistent rather than reimplemented by each application team.

Fallback is a policy, not magic

Fallback means trying another backend when the selected backend cannot complete the request. Suppose the gateway has a 1.5-second total deadline:

# A gateway fallback chain — try backends in order until one succeeds.
providers = [("primary (gpt-5)", "error 503"),
             ("fallback (claude)", "ok"),
             ("self-hosted vLLM", "ok")]

for name, status in providers:
    print(f"try {name:22s}: {status}")
    if status == "ok":
        print(f"  -> request served by {name}")
        break
    print("     failover ->")

• Gateway processing takes 20 milliseconds.
• The primary model reaches its 800-millisecond timeout.
• The fallback responds in 250 milliseconds.

The rough path is 20 plus 800 plus 250, or 1,070 milliseconds, leaving about 430 milliseconds for response handling and network variation. That request can succeed where a direct call would have failed. These are design numbers for the example, not a latency promise.

The gateway must define which failures permit failover. A malformed request, invalid API key, or disallowed model is not fixed by sending the same request to three providers. An HTTP 429, provider timeout, or temporary server error may be worth failing over. Classify these failures in policy and test the classification against each provider.

The fallback must support the same contract. Compare context-window limits, tool calls, structured output, safety behavior, language quality, and maximum output length. Every candidate must also pass the tenant and data-handling policy. If no compliant candidate remains, fail closed.

Streaming makes fallback harder. Before the first token or event, the gateway can often switch backends. After it has sent part of an answer, it cannot silently replace the rest without breaking the stream. Stop the stream, report an error, and let the client retry as a new request.

Tool calls add another edge: a provider may have returned a tool call and the executor may have run it before the gateway timed out. Retrying can execute it twice and charge a card twice. Tool executors need deduplication keys, transaction checks, or an approval step.

The useful policies live at the gateway

Routing and model aliases

Do not spread provider model names through application code. Give applications stable capabilities such as support-fast, support-reasoning, and embedding-default, then map those aliases to concrete backends.

Routing rules need an explicit reason: cost, tenant restrictions, modality, health, or request complexity. A gateway centralizes the rule but does not know whether the cheaper answer is good enough. Pair routing with an evaluation set and compare quality, refusal behavior, latency, and cost.

Limits and budgets

Request limits protect capacity; token limits protect the resource that drives many model costs. A token bucket with a capacity of 10,000 tokens and a refill rate of 2,000 tokens per minute behaves differently from 100 requests per minute. One 9,000-token document can cost more than 50 short questions.

The gateway can estimate tokens before dispatch, reserve budget, and reconcile against the provider’s final usage report. Estimates can be wrong, especially for images, cached prefixes, tool results, or provider-specific tokenization, so handle both reservation and final adjustment. Reservations must be atomic or coordinated through a shared store; a counter in each replica is not one budget.

Caching

An exact cache can return a previous answer for an identical effective request. Its key must include the concrete provider and model, route-configuration revision, response-affecting parameters, messages, tools, policy, and tenant or personalization context. Omitting tenant context can turn an optimization into a data leak.

Semantic caching returns an answer for a judged-similar request. It can work for stable factual questions but is risky for account balances, permissions, inventory, or other changing private state. Normally bypass it for tool calls and personalized requests. Caching reduces cost, provider traffic, and latency, but the trade-off is freshness: a stale answer is still wrong.

Observability

A gateway gives every request a common place to emit a request ID, trace ID, provider, model alias, status, latency, token counts, and estimated cost. Record:

• request and error rates by application, tenant, model, and provider;
• time to first token and total completion time;
• fallback reasons, token usage, cache hits, and budget or rate-limit rejections.

Do not treat gateway latency as model latency. Track gateway overhead separately. Logs also need restraint: prompts may contain names, account numbers, source code, or secrets. Store metadata by default; redact sensitive fields, restrict access, define retention, and make raw-prompt capture explicit.

Gateway or direct SDK?

The question is whether shared policy justifies another service in the request path.

A gateway adds latency, operational cost, and a blast radius. It can also become a least-common-denominator API, delaying an unusual provider capability. A direct path may be cleaner when one application needs that feature immediately. If you provide an escape hatch, keep it governed rather than copying provider keys into random services.

Failure modes you will see first

Centralization removes duplicated failure modes but creates one shared dependency. Run the data plane redundantly, keep enough configuration locally to survive a control-plane hiccup, and define the failure posture per policy. Fail-open may be reasonable for optional analytics. It is usually unacceptable for a data-residency rule or approval before a financial tool call.

Build or buy

A tiny internal proxy can be sensible for one team, two providers, and a narrow requirement such as hiding keys, applying timeouts, and emitting usage metrics. It becomes expensive when it must handle provider authentication, streaming, tools, usage accounting, rate-limit headers, retries, fallback compatibility, dashboards, and configuration rollout.

Evaluate an existing gateway before building that surface area. LiteLLM is a widely used open-source proxy with support for many providers and common routing and governance features. Portkey, Cloudflare AI Gateway, and Kong AI Gateway offer managed or platform-oriented alternatives. Check current provider coverage, caching behavior, enterprise controls, and pricing.

Test the requests that matter: streaming, tools, structured outputs, multimodal inputs, usage accounting, timeout behavior, and failure recovery. A gateway that handles ordinary text but drops a tool argument during fallback is not production-ready.

Next

The gateway is where routing, rate limiting, caching, and guardrails converge. For keeping model services healthy under real load, see LLM serving ops.

Quick check