AI gateways

Your first LLM feature calls one provider with one SDK. Then product wants a cheaper model for easy queries, compliance wants a fallback when the primary is down, finance wants per-team budgets, and security wants every prompt logged. Suddenly each service carries its own provider SDKs, its own keys, its own retry logic, its own logging — the same cross-cutting concerns reimplemented badly in a dozen places. An AI gateway fixes this the way an API gateway always has: put one proxy in front of every model call and solve those concerns once.

One chokepoint in front of every model

An AI gateway sits between your applications and all the model providers, exposing a single API — almost always OpenAI-compatible, so your code targets one format and the gateway translates to whatever backend actually serves the request. Everything that should be uniform across model calls lives in that one layer:

The concerns it centralizes are exactly the ones you have separate lessons on — because the gateway is where they all converge:

• Unified API & routing. Swap providers by config, not code (model: "anthropic/claude-…" today, "openai/gpt-…" tomorrow). It generalizes the model-routing cost cascade into one place.
• Resilience. Retries, timeouts, and fallback across providers/keys, so one provider’s outage doesn’t take you down.
• Governance. Virtual keys, per-team budgets, and rate limits enforced centrally — no app can blow the bill.
• Caching. Exact/semantic caching in one layer instead of re-implemented per service.
• Observability. Unified logs, traces, and token/cost metrics for every call — the answer to “why did the bill spike?”
• Guardrails. A natural hook point for input/output validation.

Fallback in action

The most load-bearing gateway feature is fallback: define an ordered list of backends, and the gateway tries the next one whenever the current fails (error, timeout, rate limit). The app sees one successful response; the failover is invisible.

# A gateway fallback chain — try backends in order until one succeeds.
providers = [("primary (gpt-5)", "error 503"),
             ("fallback (claude)", "ok"),
             ("self-hosted vLLM", "ok")]

for name, status in providers:
    print(f"try {name:22s}: {status}")
    if status == "ok":
        print(f"  -> request served by {name}")
        break
    print("     failover ->")

try primary (gpt-5)       : error 503
     failover ->
try fallback (claude)     : ok
  -> request served by fallback (claude)

The primary returns a 503; the gateway transparently fails over to the next backend and the request still succeeds. The application never wrote a line of retry-and- failover logic — the gateway owns it for every call.

In one breath

• Once you call more than one model/provider, cross-cutting concerns (retries, limits, caching, logging, cost, keys) get reimplemented per service; an AI gateway is one proxy that solves them once.
• It exposes a unified, usually OpenAI-compatible API and centralizes routing + fallback, retries/timeouts, rate limits + budgets, caching, observability, and guardrails.
• Fallback is the load-bearing feature: an ordered backend list means one provider’s outage transparently fails over and the request still succeeds.
• Build vs buy: LiteLLM (OSS proxy, 100+ providers) or managed Portkey / Cloudflare / Kong — treat it as infrastructure.
• It becomes a critical dependency in every call path, so run it redundantly, keep its overhead tiny, and define how it fails.

Quick check

Next

The gateway is where routing, rate limiting, caching, and guardrails converge. Keeping the whole service healthy under real load — autoscaling, cold starts, capacity, chaos — is LLM serving ops.