What is LLM model routing and how does an LLM cascade work?

How to think about it

LLM (large language model) model routing chooses the most suitable model for each request instead of sending every request to the largest model. An LLM cascade is a sequential routing strategy: try a cheaper model first, check its result, and escalate to a stronger model only when the result is not good enough.

The point is not to make a small model magically equal a large one. The point is to spend large-model budget on the minority of requests that actually need it.

Why model routing exists

A production application usually receives a mixture of easy and difficult requests. A support bot might answer “How do I reset my password?” thousands of times a day, then receive one complicated request involving a refund, a policy exception, and three previous interactions.

Using the strongest model for both requests is simple, but wasteful. The difficult request may need careful reasoning and a long context. The password question does not. If the strong model costs 30 times as much, has twice the latency, and handles 80 percent of the traffic, the simple architecture is quietly burning money.

Routing treats model choice as a decision problem. For each request, the system considers signals such as:

• the task or intent, such as classification, summarization, coding, or policy advice;
• input length and required context window;
• whether the request needs tool use or multi-step reasoning;
• language and domain;
• risk level and data-residency requirements;
• the latency budget;
• the model’s measured quality on similar evaluation examples.

A router then selects a model or model tier. The selection might come from simple rules, a trained classifier, a learned routing policy, or another language model acting as a judge. A rule such as “send requests containing a payment dispute to the strong model” is routing too. It is not glamorous, but it is often easier to audit than a mysterious neural router.

The underlying objective is usually a constrained trade-off: achieve a required quality level while reducing cost and latency. A useful way to express it is maximize quality while keeping cost and latency within budget. The weights differ by product. A gaming assistant may care mostly about speed. A medical workflow may prefer a slower, more expensive answer if it reduces dangerous errors.

How a cascade works

A cascade is routing over time rather than choosing once.

The first model produces an answer. A quality gate, meaning a check that decides whether the answer is acceptable, examines it. If it passes, the system returns it. If it fails, the request moves to a stronger model, often with the original context and the first draft available.

A cascade can have two stages or several:

1. A small, cheap model handles ordinary requests.
2. A medium model handles answers that need more reasoning.
3. A large model handles the remaining difficult or high-risk cases.

The gate is the important part. “The model says it is 95 percent confident” is not automatically a useful gate. Models can be confidently wrong, and confidence scores are often poorly calibrated. A calibrated score of 0.8 would mean that roughly 80 percent of cases receiving that score meet the agreed quality definition. You must test that relationship rather than assume it.

Better gates use task-specific evidence:

• Did the answer follow the required JSON schema?
• Did it cite a retrieved policy section?
• Did it extract all required fields?
• Did a deterministic calculation validate?
• Did a separate verifier detect unsupported claims?
• Did the answer satisfy a business rule?
• Did the model abstain when the source material did not contain an answer?

The gate should measure the thing the product cares about. A generic “confidence” number is usually a weak substitute.

A concrete cost example

Imagine a bank support bot with three available models. These are illustrative per-request costs for one fixed prompt and response size, not universal provider prices:

Model	Typical use	Cost	Latency
Small	FAQ and simple extraction	$0.001	250 ms
Medium	Policy reasoning	$0.006	700 ms
Large	Ambiguous or high-risk cases	$0.030	2 seconds

Suppose 10,000 requests arrive in one day:

• 8,000 answers pass the small-model gate.
• 1,500 fail the small gate but pass with the medium model.
• 500 fail both gates and reach the large model.

Every request pays for the first call. The daily model cost is therefore:

• 8,000 small-only requests: $8.00
• 1,500 small-plus-medium requests: 1,500 × ($0.001 + $0.006) = $10.50
• 500 requests reaching all three models: 500 × ($0.001 + $0.006 + $0.030) = $18.50

The total is $37.00, or $0.0037 per request on average. Sending all 10,000 requests directly to the large model would cost $300.00 under the same assumptions. That is about 88 percent more.

The cascade does not make the difficult requests cheap. Those 500 requests still pay for every attempt. It makes the common path cheap.

There is also a subtle comparison here. If a perfect one-shot router could identify the final model in advance, it would cost only $32.00: 8,000 small calls, 1,500 medium calls, and 500 large calls. The cascade costs $37.00 because escalated requests pay for earlier attempts. In practice, the perfect router does not exist, and the cascade often gives a safer quality path.

A simplified implementation might look like this:

def answer(ticket, policy_context):
    draft = call_model("small", ticket, policy_context)

    if passes_gate(draft, ticket, policy_context):
        return draft

    draft = call_model("medium", ticket, policy_context)

    if passes_gate(draft, ticket, policy_context):
        return draft

    return call_model("large", ticket, policy_context)

call_model and passes_gate here are policy pseudocode, not a provider-specific API. In production, the system would also record the selected tier, token usage, gate decision, latency, and final outcome.

The senior-level nuance

“Use the smallest model first” is a useful default, not a law.

A small specialized classifier may outperform a larger general model on intent classification. Conversely, a request involving an irreversible action, sensitive personal data, or a complex legal policy may deserve the strong model immediately. A cascade is a poor safety boundary if the first model can trigger a payment, delete data, or send an external message. Keep generation separate from authorization, and require deterministic checks or human approval before side effects.

Cascades also add latency. In the example, a request reaching the large model may wait roughly 250 ms + 700 ms + 2 seconds, before network overhead and gate time. If the small model starts failing often, the application pays for several calls and users experience a slow tail. When most traffic reaches the final stage, calling the large model directly may be both cheaper and faster.

The router itself can become a failure point. A prompt change may cause the classifier to label more requests as “complex,” increasing spend without improving answers. A new model may change formatting and make an otherwise correct answer fail a strict schema gate. A user can also deliberately phrase a request to influence routing. Routing, tool authorization, and safety policy should therefore be separate decisions.

The first production symptom is often not an obvious quality failure. It is a rising fallback rate, token bill, or p95 latency. For example, if fallback rate rises from 12 percent to 38 percent after a prompt update, inspect the gate before blaming the models. The gate may now expect a field name the model no longer emits. Conversely, a gate that accepts almost everything can leave the cost graph looking healthy while unsupported answers quietly reach customers.

Evaluate the whole policy, not just each model. Build a test set containing easy, hard, ambiguous, multilingual, long-context, adversarial, and high-risk requests. Measure:

• final answer quality;
• false acceptance, where a bad first answer passes;
• false rejection, where a good first answer escalates;
• cost per request;
• latency by route and percentile;
• fallback rate;
• quality by user segment and task type.

The threshold should be chosen from those measurements. There is no magic value such as “accept when confidence exceeds 0.8.” A false acceptance may be much worse than a false rejection for a bank dispute, while the opposite may be true for a low-stakes brainstorming tool.

A robust cascade also gives every stage the same authoritative context. If the small model sees one retrieved policy passage and the large model sees a different one, escalation may appear to fix the answer when it actually changed the evidence. Log the context version, model version, prompt version, and gate version so an incident can be reconstructed rather than guessed at.

What they’ll ask next

Is a cascade the same thing as a fallback?
Not exactly. A fallback usually means “try another service when the first one errors.” A cascade escalates because the first response may be syntactically valid but insufficient in quality. A timeout fallback is operational resilience; a quality-gated cascade is cost and quality management.

How do you know whether the first answer is good enough?
Use a task-specific gate: schema validation for structured output, retrieval support for grounded answers, tests for code, and a calibrated or separately evaluated verifier where necessary. Do not rely only on the model’s self-reported confidence.

How would you evaluate a router?
Compare it with always using the large model and with simpler rules. Report final quality, cost, latency, fallback rate, and false acceptances on a representative held-out set. The router wins only if its savings do not violate the product’s quality or safety floor.

For the deeper mental model and implementation patterns, see the model routing lesson.

Say this in the interview: “Model routing chooses the right model per request; a cascade is a sequential router that starts cheap and escalates when a tested quality gate rejects the result, trading a little extra latency on hard cases for much lower average cost.”