When much of your traffic is easy: use routing to cut LLM costs

At 9:07 on a Monday, finance opens the LLM invoice and finds that a support assistant spent $3,000 last month answering questions like “Where is my invoice?” and “How do I reset my password?”

The assistant is powered by a frontier model, meaning a highly capable general model that is usually the most expensive option. It was chosen during a product demo because the hard questions looked impressive. Then production arrived. Seventy-five thousand routine requests got exactly the same treatment as the difficult ones.

That is not really a model-quality problem. It is an allocation problem.

Most teams do not need to make every model call cheaper. They need to stop paying frontier prices for work a cheaper model can complete correctly. Route the easy slice down. Keep the difficult slice on the strong model. Cache safe repeats before either model sees them.

The important word is correctly. Routing is not a license to send random requests to a tiny model and celebrate the lower invoice. A cheap wrong answer is often an expensive support ticket wearing a disguise.

The bill follows traffic distribution

Build a baseline

Suppose that support assistant handles 100,000 requests each month. For illustration, use these blended costs:

• the cheap model costs $0.002 per request;
• the frontier model costs $0.03 per request;
• the router costs $0.0005 per request.

These are scenario numbers, not a provider price list. Real costs depend on input tokens, output tokens, context length, cached tokens, and the provider’s pricing model. Use your own usage logs when you calculate the business case.

If every request goes to the frontier model, the monthly model cost is:

100,000 × $0.03 = $3,000

Compare the routes

Now suppose the traffic really does split into 75,000 easy requests and 25,000 hard ones. A router sends the easy requests to the cheap model and the rest to the frontier model.

Path	Requests	Cost per request	Monthly cost
Cheap model	75,000	$0.002	$150
Frontier model	25,000	$0.03	$750
Router	100,000	$0.0005	$50
Total			$950

In this illustrative 75%-easy scenario, routing costs 68.3% less than sending all 100,000 requests to the frontier model, before changing prompts, providers, or infrastructure. That percentage is not a general routing range.

Savings are workload-specific: the result changes with the easy fraction, model prices, router cost, cache hit rate, and quality threshold.

Measure the easy slice

The arithmetic is simple. The engineering is deciding whether a request is safe to route down.

A useful model is:

C_route = N [p c_C + (1 - p)c_F + c_R]

Here, N is the number of requests, p is the fraction sent to the cheap model, c_C is the cheap-model cost, c_F is the frontier-model cost, and c_R is the routing cost.

The savings come from three things:

• a large cheap fraction;
• a meaningful price gap;
• a router whose own cost is small.

If only 10% of traffic is easy, or if the two models cost nearly the same, routing will not perform financial magic. It will merely add another component to your architecture.

“Easy” also does not mean “short.” A 200-word request to format a list may be trivial. A 12-word question about whether a customer can cancel a contract under an exception clause may be difficult. Token length predicts part of the bill. It does not reliably predict the reasoning or risk required.

A router estimates capability, not importance

Model routing is the act of choosing a model for each request instead of using one model for everything. A router is the component that makes that choice. It can be a rules engine, a learned classifier, or another language model instructed to classify the request.

Define the acceptance test

The router’s real question is not “Is this prompt interesting?” It is:

Can the cheaper candidate satisfy this task’s acceptance test?

That test might be exact classification accuracy, valid JSON, a correct SQL query, a citation supported by retrieved text, a passing unit test, or a human reviewer’s judgment. Without an acceptance test, “easy” becomes a feeling. Feelings make poor FinOps infrastructure.

Start with observable signals

A first router can use obvious signals:

• intent, such as password reset versus contract interpretation;
• whether tools or multiple tool calls are required;
• whether the request asks for transformation or new reasoning;
• whether retrieved documents conflict;
• whether the tenant or account context makes the answer personalized;
• whether the request resembles a known class of past failures.

A learned router can go further. It can learn from preference data, meaning examples where one model’s answer was judged better than another’s. RouteLLM, developed by researchers at UC Berkeley with industry collaborators, is a well-known example of this approach.

But a routing score is not a prophecy.

The threshold is where your business decision lives. Set it too low and the cheap model receives hard requests. Set it too high and the frontier model receives nearly everything, which is safe but financially pointless.

RouteLLM’s published result shows why routing is worth taking seriously, but also why benchmark numbers need their labels attached. In one reported MT-Bench configuration, RouteLLM retained about 95% of GPT-4’s performance, reduced estimated cost by about 85%, and needed GPT-4 for roughly 14% of calls after data augmentation.

That 85% figure is an evaluation result for that benchmark configuration, not a general production savings range. It is evidence that a router can work. It is not a guarantee that your support bot, coding agent, or legal workflow has the same traffic distribution or quality curve.

The model routing lesson goes deeper into router designs and threshold selection. The practical point is shorter: measure the quality curve for your own requests.

Cascades trade latency for a second chance

A cascade is a related pattern with a different shape. Instead of deciding everything before generation, it tries the cheap model first, checks the result, and escalates to the frontier model when the check fails.

That check matters. “Escalate when confidence is low” sounds tidy, but a model’s self-reported confidence is not a reliable quality monitor by default. Better checks are tied to the task:

• Did the generated JSON parse?
• Did the SQL execute and return a sensible schema?
• Did the answer cite a retrieved passage that actually supports it?
• Did the tool call satisfy the required argument constraints?
• Did a deterministic test pass?

For open-ended answers, a second model can judge the first, but that judge is itself fallible. Sampled human review and task-specific evaluations remain necessary.

Using the support example, a cascade sends all 100,000 requests to the cheap model at $0.002 each. If 25,000 difficult requests fall through to the frontier model, the base cost is:

100,000 × $0.002 + 25,000 × $0.03 = $950

That is the same model arithmetic as direct routing, before paying for a verifier. The cascade has one advantage: it can discover difficulty from the generated answer.

It has two disadvantages:

• every request pays for the first generation;
• hard requests wait for two model calls.

Direct routing is usually better when you can classify difficulty reliably before generation. Cascades are useful when the failure is visible only after generation. A code-generation task with a test suite is a good cascade candidate. A customer asking a sensitive policy question may not be, because a plausible but wrong answer can pass a superficial check.

Cache before you route

Routing avoids an expensive call. Semantic caching avoids the generative-model call entirely when a new request means approximately the same thing as one seen before.

An exact cache matches a normalized request byte for byte. A semantic cache converts text into an embedding, a numeric representation of meaning, and retrieves nearby previous requests. “How do I reset my password?” and “I forgot my password; how can I get back in?” may be close enough to share an answer.

A semantic-cache lookup is not free. The new request still needs embedding generation and a vector lookup, even when the response is a hit. A hit skips the expensive generative-model call, not every form of inference.

Check reuse before model selection

The order should usually be:

1. exact cache;
2. safe semantic cache;
3. router;
4. cheap or frontier model;
5. validation and cache write.

In the running example, imagine a 30% cache-hit rate. Assume the cache’s amortized infrastructure cost is $0.0001 per incoming request. This estimate includes embedding generation, vector-database operations, cache writes, and cache-side work for misses; it excludes the router and model calls, which are counted separately below.

The remaining 70,000 requests have the same 75% easy and 25% hard split:

• cache infrastructure: 100,000 × $0.0001 = $10;
• cheap model: 52,500 × $0.002 = $105;
• frontier model: 17,500 × $0.03 = $525;
• router on misses: 70,000 × $0.0005 = $35.

Under this specific 30% cache-hit plus 75%-easy-on-misses scenario, total cost is $675, or 77.5% below the $3,000 frontier-only baseline.

Do not add a 30% cache saving to a 68.3% routing saving. The layers apply to different traffic. Calculate the residual each time.

Semantic caching is also the easiest place to create a quiet security incident.

For safe cache entries, include these in the cache key or validation rules:

• the relevant tenant and permission scope;
• prompt version;
• model version;
• locale;
• retrieved-document version;
• product state.

Add a time-to-live and invalidate entries when policies or source documents change.

A public, versioned password-reset article is a reasonable semantic-cache candidate. “What is the balance on my account?” is not. The latter is not merely a question about language. It is a live authorization decision.

The failure modes are predictable

Misrouting hard requests

The first failure looks like a success: the cheap-model percentage rises, latency improves, and the bill falls. A week later, support agents notice that short legal or billing questions are getting confident nonsense.

The router learned that short requests were easy because length was an easy feature. The symptom is not an exception in your logs. It is a slow rise in human corrections and user rephrasing.

Fix it with hard negatives:

• short requests that require strong reasoning;
• long requests that are genuinely easy;
• examples from every serious incident.

Track the false-cheap rate, meaning the fraction of requests sent to the cheap model that should have used the frontier model. A low fallback rate is not good if the fallback detector cannot see wrong answers.

Stale answers

The second failure is stale cache data. The assistant starts saying that a plan includes a feature removed three weeks ago. Or one customer’s wording retrieves an answer generated for a different plan tier.

The first symptom is often a complaint that seems unrelated to the cache. Log these fields:

• the cache key;
• matched source;
• similarity score;
• tenant scope;
• document version;
• final answer.

Without those fields, debugging becomes archaeology.

Costs hidden by request counts

The third failure is that the bill barely moves. The router dashboard says 80% cheap traffic, but the provider invoice is almost unchanged.

Common causes include:

• very long prompts;
• large generated answers;
• retries;
• verifier calls;
• frontier fallbacks hidden behind an application-level “single request.”

Measure input and output tokens by route. A cheap model processing a 50,000-token retrieved context may be cheaper than the frontier model, but not as cheap as the dashboard’s per-call label suggests.

The LLM cost and latency guide is useful here because the denominator must be actual tokens and completed work, not just request count.

The strongest objection is reasonable

The best argument against routing is not that it cannot save money. It is that it adds a new failure plane.

A single strong model is easier to test. It has one prompt, one set of behavior changes, and one set of operational dashboards.

The failure plane can include:

• a router misclassifying a request;
• a cascade adding latency;
• a verifier approving a bad answer;
• a semantic cache returning something stale.

If the cheap model fails often enough, the retries erase the savings.

That objection wins in some environments.

Do not route a low-volume workflow whose bill is immaterial. Do not use a semantic cache for rapidly changing, personalized data. Do not send a high-stakes action to a weaker model merely because the prompt looks simple. And if nearly every request needs multi-step reasoning, tool use, or careful evidence synthesis, the easy fraction may be too small to justify the machinery.

For other workloads, the answer is disciplined rollout rather than blind uniformity. Keep the frontier model for critical intents. Shadow the router before it changes production answers. Use deterministic validators where possible. Make the fallback path explicit. Evaluate quality by task slice, not only by an overall average.

The right metric is cost per successfully completed task. In the earlier example, the direct route cost $950. If 8% of the 75,000 cheap calls fail and are detected, 6,000 frontier retries add another $180. The total becomes $1,130.

That is still a 62.3% reduction, but only if the failures are detected and the user receives a correct result. If the failures are invisible, the cost number looks better while the product gets worse.

LLM evaluations should be part of the routing system, not a one-time launch ritual.

What to do on Monday morning

Start with a ledger, not a router.

1. Export the last 7–14 days of requests. Keep:

   • the request;
   • retrieved context;
   • tool calls;
   • input and output tokens;
   • selected model;
   • latency;
   • retries;
   • user feedback;
   • final outcome.

   Redact sensitive values before using the data for evaluation. Group requests by intent, tenant type, tool use, and failure history.

2. Define success for each important intent.

   • For password resets, success may be a correct structured action with no unauthorized account change.
   • For policy questions, it may be an answer supported by the current policy document.
   • For classification, it may be an exact label.

   Use human review where no deterministic test exists. A small, carefully labeled set of 300 examples is more useful than a large pile of unlabeled prompts.

3. Replay the same requests against the current frontier model and one cheaper candidate. Keep the prompt and retrieved context fixed. Compare:

   • pass rate;
   • harmful-error rate;
   • output length;
   • latency;
   • cost.

   Slice the results by intent and difficulty. The average can look excellent while one important customer segment quietly suffers.

4. Begin with obvious rules and shadow routing. Route stable transformations and simple classifications down first. Let the shadow router make decisions in logs while production still uses the existing model. Compare its proposed route with the measured outcome. Do not tune a threshold from intuition.

5. Add exact caching before semantic caching. Exact hits are easy to reason about. For semantic caching, start with public, versioned, non-personalized answers. Record every match and manually inspect false positives. Set a conservative similarity threshold from observed errors, not from a library default.

6. Canary the change and predeclare rollback conditions. A canary can expose:

   • latency;
   • provider errors;
   • cache mistakes;
   • unexpected fallback volume.

   These issues can appear before the whole workload moves. Choose quality limits with the product owner. For an action-taking workflow, the allowed regression may be zero even if a small regression is acceptable for a drafting assistant.

7. Watch the complete path. Track:

   • cheap-model share;
   • frontier share;
   • fallback rate;
   • cache-hit rate;
   • false-cheap rate;
   • token spend;
   • 95th-percentile latency;
   • user rephrases;
   • human corrections;
   • cost per successful task.

   A green “cost per request” chart is not enough.

The gateway pattern helps once more than one application needs this logic. An LLM gateway—a shared proxy between applications and model providers—can centralize:

• routing;
• caching;
• budgets;
• retries;
• provider failover;
• usage logs.

It should centralize policy, not hide it. Every route still needs an explanation that an engineer can inspect after the 3 a.m. page.

Most LLM traffic does not deserve the same amount of computation. That is the durable idea. Routing captures it without pretending that difficulty is obvious, confidence is calibrated, or semantic similarity equals sameness.

Measure the easy fraction. Protect the hard fraction. Cache only what is safe to reuse. Then let the invoice reflect the work the model actually had to do.

For the implementation details, see the model routing lesson and the caching lesson.