Most of your traffic is easy: cut LLM bills 40–85% with routing
Real query traffic is mostly simple, yet teams send everything to the most expensive model. Routing easy queries to cheap models, escalating only the hard ones, and caching repeats captures most of the quality at a fraction of the cost.
Here’s a number that should bother anyone with an LLM bill: most of the queries you send to your most expensive model don’t need it. Greetings, simple lookups, short classifications — they’d be answered correctly by a model a fraction of the price. Yet the default architecture routes everything to the frontier model, like taking a taxi to your mailbox. Model routing fixes that, and the savings are large enough to change a product’s unit economics.
Route easy queries down, escalate the hard ones
The idea is simple: estimate each query’s difficulty and send it to the cheapest model that can handle it, reserving the expensive model for the genuinely hard slice. RouteLLM, from UC Berkeley with Anyscale and Canva and published at ICLR 2025, trains routers from preference data to do exactly this. The dilemma it resolves is the one every team faces: route everything to a strong model and you get top quality at inflated cost; route to a weak model and you save money but lose quality — a good router beats both. And the results are not marginal: RouteLLM kept 95% of GPT-4’s performance on MT-Bench while cutting cost about 85%, and with data augmentation needed to call GPT-4 for only ~14% of queries. The full decision framework — routing versus cascades, where the sweet spot sits — is in the model routing lesson.
Cascades and caching stack on top
Routing has a sibling: the cascade. Try a cheap model first, verify the answer, and only fall through to an expensive model when confidence is low — since most queries pass the cheap tier, you pay frontier prices only for the residual. Layer on semantic caching, which returns a cached answer when a new prompt means the same thing as a previous one, even if it’s worded differently, and you skip the model entirely for repeats. The combined effect is substantial: routing plus cascades commonly cut costs 45–85% while keeping ~95% of quality, and adding semantic caching pushes total savings past 60% on high-repetition workloads.
It’s production infrastructure now
This stopped being a research trick. Red Hat shipped an open-source LLM Semantic Router for complexity-based request routing, and the practical home for routing, caching, failover, and cost tracking is an LLM gateway that sits between your app and the providers — so you get the savings without rewriting application code.
The headline is freeing: you don’t have to choose between quality and cost. Because the hard queries are a small minority, routing just that minority to the frontier model captures almost all the quality for a fraction of the spend. The model routing lesson and the caching lesson walk through how to build it.