Most companies already using generative AI send every task to the same model: the most capable one they tried on day one. Classifying a three-line email and drafting a risk report go through the same pipe and cost the same. Model routing fixes that: each task goes to the model that fits it on capability and cost.
What model routing is
A layer that decides, on each request, which model handles it. The decision can be based on the type of task, the length and complexity of the input, how demanding the output is, or explicit business rules — for example, that anything touching customer data goes to a specific deployment.
A common setup has three lanes: a small, cheap model for classification, extraction, and routine answers; a mid-tier model for drafting and summarizing; and a top-tier model reserved for complex reasoning, code, or sensitive decisions. Most of the traffic in a typical business application doesn’t need the expensive lane.
The three pieces that make it work
Routing. The rule that maps tasks to models. It can start as an explicit list by operation type and evolve into a classifier that estimates difficulty before deciding.
Fallback. What happens when the chosen model fails, takes too long, or returns something unusable: retry, move up a lane, or degrade gracefully with a warning. Without fallback, your service is only as available as your provider.
Caching. Many requests repeat word for word, or nearly. Storing and reusing responses — and, where the provider allows it, the context itself — means you don’t pay twice for the same thing. It’s the part with the best effort-to-savings ratio.
What you gain beyond cost savings
Independence. An application with routing is already set up to talk to more than one provider. Switching, spreading load, or negotiating stops being a project.
Latency. Short tasks handled by small models come back much faster. In a user-facing assistant, that’s more noticeable than marginal gains in text quality.
Risk control. Routing rules are also governance rules: which data can go to which provider, and what stays in a controlled deployment.
When it makes sense — and when it doesn’t
It makes sense when:
- Monthly model spend is already a visible line item and grows with usage
- Very different tasks live in the same application
- Availability matters: if your single provider goes down, your service stops
- There are requirements about where each type of data can be processed
It doesn’t make sense when:
- Volume is low: below a certain spend, the added complexity costs more than it saves
- There’s only one type of task and it’s demanding: if everything needs the expensive lane, routing adds nothing
- There’s no way to measure quality: without evaluation, moving to a smaller model means lowering quality blindly
How we approach it
First, we measure: what types of request there are, in what proportion, and what each one costs today. Then we define the routing matrix and set the quality bar for each lane with a custom evaluation set. Only then do we move to smaller models where the results hold up, and keep the rest on top. We report savings against last month’s actual bill, not against an estimate.
We have no vendor agreements: the routing matrix answers to the use case, not to anyone’s catalog.
Frequently asked questions
Doesn’t moving to a smaller model hurt quality? On demanding tasks, yes — which is why we don’t do it everywhere. The work is identifying where the small model’s output is indistinguishable, and measuring that before switching.
Do we need contracts with several providers? Not necessarily: almost all of them offer several model sizes. Working with more than one adds resilience and negotiating leverage, but you can start within a single model family.
How much will we save? It depends on your task mix and your starting point. The useful number isn’t a generic percentage — it’s the before-and-after bill at the same volume and the same measured quality.