Model routing.
September 26, 2026
Most production traffic doesn't need the frontier model. Routing easy queries to a smaller/cheaper model and hard queries to the frontier model is the single highest-leverage optimization in the playbook ; typically 30–50% reduction with no quality regression, when you do it right. Here's how we do it.
The two routing patterns that actually work
1. Classifier-up-front
A small classifier (often a fine-tuned `gpt-5-mini`, `claude-haiku`, or a local distilled BERT) reads the incoming request and predicts difficulty. Easy → cheap model. Hard → frontier model. Edge cases → frontier model.
Works well when your traffic has visibly distinct difficulty buckets (e.g. customer support: simple FAQ vs. multi-turn troubleshooting). Doesn't work when difficulty is uncorrelated with surface features.
2. Cascade with confidence
Cheap model attempts the answer. If its self-reported confidence (or a verifier check) is below threshold, escalate to the frontier model. Pay the cheap-model token cost twice for ~10–20% of traffic; pay the frontier-model cost on a fraction of overall traffic.
Works well when the cheap model is right most of the time and confidence is well-calibrated. Don't trust raw `logprobs` as confidence on aligned chat models ; train or distill a verifier.
Patterns that don't work
- Round-robin or weighted random across models of different quality. You'll meet your savings target and miss your quality SLO.
- "Pick the cheapest model that passed eval" as a static decision. Real traffic distribution drifts away from your eval set within weeks.
- Routing on token count alone. Long doesn't mean hard. Short doesn't mean easy.
Quality measurement
Every router ships with a 7-day A/B against the production baseline. Quality SLOs are agreed per endpoint:
- LLM-as-judge on a held-out eval set, scored against the frontier-model output as reference.
- Production proxies ; thumbs-up rate, conversation length, escalation rate, downstream task completion.
- Human spot-checks on 50–200 sampled responses per week.
If any SLO degrades meaningfully, the router auto-rolls back. There's no shipping a router without a rollback path.
Stack
- Gateway with routing logic: LiteLLM, Helicone, or a custom proxy. We tend toward custom when the routing rules are non-trivial.
- Classifier: fine-tuned mini-model on the platform you're already on (avoids adding a new vendor).
- Verifier for cascades: distilled BERT or a fine-tuned mini-model trained on your eval traces.
- A/B framework: anything that can split by user/session and persist assignment for at least 7 days.
Order of operations
- Identify the top 3 spend endpoints.
- For each: build a 200-row eval set with frontier-model answers as reference.
- Test 2–3 candidate cheaper models against the eval set. Keep the ones that pass quality.
- Build the classifier or cascade on the candidates that passed.
- Ship at 1% traffic. Then 10%. Then 50%. Then 100%. A/B at every step.
Related
- Built-in tools cost economics
- Full LLM cost optimization guide
- Semantic caching: when it works and when it doesn't
FAQ
What is model routing for LLM cost optimization?
Model routing sends easy requests to a cheaper model and hard requests to a frontier model. The goal is to reduce cost without quality regressions by matching model strength to task difficulty. It works when routing decisions are measured against production quality, not just benchmark scores.
Which model routing patterns actually work?
Two patterns are reliable: classifier-up-front and cascade with confidence. A classifier predicts whether the request is easy or hard before routing. A cascade lets a cheaper model try first, then escalates to the frontier model when confidence or a verifier check is low.
What model routing patterns should I avoid?
Avoid round-robin or weighted random routing across models of different quality, static cheapest-model decisions based only on eval pass rates, and routing on token count alone. Those approaches can meet a savings target while quietly missing the quality SLO.
How do I ship model routing safely?
Start with the top spend endpoints, build an eval set, test cheaper model candidates, and then ship through staged production traffic: 1 percent, 10 percent, 50 percent, then full rollout. Run A/B checks at every step and keep an automatic rollback path if quality degrades.