Design a model cascade for a customer-support workload with strict cost and quality targets. What signals would you use to route requests, how would you avoid routing errors caused by overconfident small models, and how would you evaluate the cascade rather than each model in isolation?

How to think about it

I would put a cheap triage layer in front of a small responder, and send high-risk, ambiguous, or out-of-distribution requests to a stronger model or a human; routing would use calibrated risk and estimated task difficulty, not the small model’s stated confidence. I would tune the whole policy against end-to-end quality, cost, latency, safety, and resolution targets, with an explicit abstain path when the cheap route is not safe.

Why a cascade works

Imagine the 3 a.m. queue. A customer asks, “How do I change my notification settings?” Another says, “You charged me twice; reverse one payment.” Both are seven words long. Their consequences are not remotely equal.

A model cascade is a sequence of increasingly expensive handlers. A cheap gate first decides what kind of request it is. A small model may answer routine, low-risk questions. A stronger model, deterministic workflow, or human handles requests where mistakes are expensive.

The key is that routing is a decision under unequal loss. Sending an easy FAQ to an expensive model wastes money. Sending an account-deletion request to a cheap model risks an irreversible mistake. The best route is therefore not simply the model with the lowest estimated error. It is the cheapest route that satisfies the quality and safety requirement for that particular request.

I would make the route depend on five groups of signals:

• Intent and risk: FAQ, shipment status, billing dispute, cancellation, account deletion, fraud, or regulated content. Risk includes both the chance of an error and the damage if it happens.
• Required capability: Does the answer need an authenticated account lookup, a tool call, a policy citation, structured output, or a long conversation history?
• Ambiguity: Missing order numbers, contradictory details, vague pronouns, or a request with several plausible interpretations.
• Evidence quality: Whether retrieval found an authoritative, current article that actually supports the proposed answer.
• Operational context: Language, context length, latency budget, model availability, and current queue load.

Some of these signals are available before generation. Others appear only after a draft exists. That usually leads to a two-stage cascade: pre-route obvious high-risk work, then verify the small model’s draft before allowing it to reach the customer.

A concrete cascade

Suppose a support operation handles 100,000 chats per month. These are design assumptions, not a promised benchmark:

• 65 percent are low-risk, authenticated FAQ or status questions.
• 25 percent are ordinary but ambiguous or multi-step.
• 10 percent involve financial changes, fraud, account deletion, or policy exceptions.

The policy might look like this:

Route	Example	Action
Green	“Where can I change email alerts?” with a strong knowledge-base match	Small model answers from retrieved evidence
Amber	“My refund still hasn’t arrived” with missing order details	Small model drafts; verifier checks evidence and policy; escalate on failure
Red	“Delete my account,” fraud allegation, or payment reversal	Strong model plus an authenticated workflow, or human review

Assume the fully loaded internal inference costs are 0.0002 dollars for triage, 0.002 dollars for a small response, 0.004 dollars for verification, and 0.020 dollars for a strong-model response. Those figures are illustrative; the important part is doing the accounting explicitly.

For 100,000 chats:

• Triage costs 20 dollars.
• Green handling costs 65,000 times 0.002, or 130 dollars.
• Amber handling costs 25,000 times 0.006 for drafting and verification, or 150 dollars. If verification rejects 40 percent, the fallback adds 10,000 times 0.020, or 200 dollars.
• Red handling costs 10,000 times 0.020, or 200 dollars.

The total is about 700 dollars, or 0.007 dollars per chat, before human labor and other infrastructure. If the quality target is a critical-error rate under 0.2 percent, that target must be checked separately for each route. A cheap average is meaningless if the red route is unsafe.

Notice that “refund status” is not automatically a small-model task. The model may need an account lookup, payment-system state, and a promised time window. The safe answer might be a tool-driven workflow, not a fluent paragraph.

Do not trust a small model saying “I am sure”

A small generative model can be confidently wrong because fluency and correctness are different things. Its token probabilities measure how likely it found a sequence of words, not whether those words match the company’s current refund policy.

A better approach uses calibration, meaning that a score of 0.8 corresponds to roughly an 80 percent chance of passing a defined quality test on held-out data. I would train or tune the gate on labeled support examples, then calibrate its scores on data it did not train on. The label should be operational: “Would this response pass the policy rubric and solve the customer’s issue?” It should not be “Did the answer sound plausible?”

The gate should be allowed to abstain, meaning it declines to use the cheap route. Choose the abstention threshold from a risk-coverage curve: as the system handles a larger fraction of traffic cheaply, how quickly does error rise? Set the threshold where the quality floor still holds. High-risk intents can have a much stricter threshold, or no small-model coverage at all.

For generated answers, I would combine several checks:

1. Does the request belong to an allowed low-risk intent?
2. Did retrieval return authoritative evidence?
3. Does the draft claim anything unsupported by that evidence?
4. Did the model request a tool, and does the tool result agree with the answer?
5. Did a separate verifier detect a policy violation, missing qualification, or unresolved ambiguity?
6. Is the output valid structured data when the downstream workflow requires it?

The verifier should not simply ask the same small model, “Are you correct?” That produces correlated errors: two components can share the same blind spot. Use deterministic checks where possible, and validate the verifier itself. For high-impact actions, require a strong model, a fixed business rule, or human approval regardless of model confidence.

A practical failure mode is easy to spot: the small model’s acceptance rate rises, cost falls, but repeat contacts and corrections rise for one intent such as international refunds. That is usually a calibration or drift problem, not evidence that the cascade improved. Check the route’s error rate by intent, language, product, and policy version.

Evaluate the cascade as a policy

Evaluating each model in isolation misses the central question: which requests did the model receive, and what happened after its decision?

I would build an offline replay set from historical conversations, with labels for correctness, policy compliance, resolution, escalation, and critical harm. I would stratify it by intent, language, conversation length, customer journey, and risk. For each request, run the candidate routing policy and record the complete path: triage, small-model generation, verifier, fallback, tool calls, and final answer.

The main metrics would be:

• Critical-error rate: unsafe or materially harmful responses. This gets a hard ceiling, not an average target.
• Quality pass rate: fraction of answers meeting the rubric for correctness, evidence, tone, and required next steps.
• Resolution or containment rate: issues solved without a repeat contact or unnecessary human handoff.
• Cost per conversation: every model call, verifier call, retrieval operation, and fallback.
• Latency: especially p95, because one slow fallback can ruin the customer experience.
• Coverage: fraction handled by the small route, and fraction abstaining or escalating.
• Calibration: whether predicted risk matches observed failure rates.
• Slice performance: whether one language, product, or customer journey quietly violates the target.

A historical replay has a trap: it only shows outcomes for the route that was originally chosen. To estimate alternatives, generate and evaluate candidate responses for a labeled sample, run the cascade in shadow mode, and then test the policy on a controlled production holdout. Use human review for consequential quality judgments. An LLM judge can help sort large volumes, but it is a measurement aid, not ground truth.

I would compare the cascade with baselines on the same traffic distribution: always-small, always-strong, and the existing rules. The useful output is a cost-quality frontier, not a single accuracy number. If the cascade saves 35 percent but increases critical errors from 0.1 percent to 0.4 percent, it failed a strict-quality requirement.

The trade-off that earns the senior signal

A cascade is not automatically cheaper. If every request gets a small draft, a verifier, and then a strong-model retry, it can cost more and take longer than using the strong model once. The gate must be cheap, the abstention decision must be useful, and the fallback rate must be measured.

Nor should traffic be forced into a model route during an outage or a knowledge-base failure. A circuit breaker should stop low-confidence answers, use a safe template, queue the request, or hand it to an agent. Saving 0.02 dollars during a broken payment integration is a poor bargain.

The production pattern is therefore conservative: automate low-risk, well-evidenced work; verify the boundary cases; make irreversible actions deterministic; and keep a visible escape hatch.

What they’ll ask next

How would you choose the confidence threshold?
I would calibrate it on a held-out set, select it from the risk-coverage curve, and enforce separate floors for high-risk intents and important slices. I would retune after policy, product, or traffic changes.

What if there are not enough labeled outcomes?
Start with a rubric and human labels on a stratified sample, use shadow traffic and repeat-contact signals, and treat automated judges as triage rather than truth. Label the errors that matter most first.

What happens when the strong model is unavailable?
Do not silently downgrade red-route requests. Use deterministic workflows or safe holding responses, queue for a human, and alert on the circuit breaker. Availability is part of the cascade’s quality contract.

One line to say in the room

“I would optimize a calibrated, abstaining routing policy for end-to-end risk-adjusted cost, then prove it on the traffic slices and failure paths customers actually experience—not on isolated model accuracy.”