One model for everything is wasteful
Using your most powerful (and expensive) model for every request is like taking a freight truck to buy groceries. Most queries — simple classifications, short rewrites, FAQ answers — are handled perfectly by a small, fast, cheap model. Model routing sends each request to the *cheapest model that can handle it*, cutting cost and latency without hurting quality on the hard cases.
How routing works
A router sits in front of your models and decides where each request goes, based on signals it can read from the request:
{
"rules": [
{ "if": { "task": "classify" }, "use": "claude-haiku-4-5" },
{ "if": { "task": "summarize", "tokens_lt": 2000 }, "use": "claude-haiku-4-5" },
{ "if": { "task": "reason" }, "use": "claude-opus-4-8" }
],
"default": "claude-haiku-4-5"
}Rules expressed as JSON are easy to read, version, and adjust without code changes.
Three routing strategies
| Strategy | How it decides | Best for |
|---|---|---|
| Rule-based | Task type, length, flags in the request | Predictable workloads |
| Classifier | A tiny model predicts difficulty | Mixed, unlabeled traffic |
| Escalation | Try cheap model; verify; retry on the big one if it fails | Maximizing savings safely |
The escalation pattern is especially effective: run the cheap model, validate its output (schema check, confidence, self-rating), and only fall back to the expensive model when the cheap one falls short.
Escalation in practice
1. Send request to the cheap model.
2. Validate the JSON output against the schema.
3. If valid and confident -> done (cheap path).
4. If invalid or low-confidence -> retry on the strong model.Because most requests succeed on the cheap path, your average cost drops toward the cheap model's price while quality stays at the strong model's level for the hard cases.
Routing signals worth capturing
- Task type — classification vs reasoning vs generation.
- Input size — long contexts may need a bigger model (or a cheaper long-context one).
- Required quality — a draft can use a small model; a customer-facing answer may not.
- Latency budget — interactive UIs favor fast models.
Pass these as structured fields on the request so the router can read them without guessing.
Measure before and after
Routing is an optimization, so measure it. Log which model handled each request, the cost, and a quality signal (validation pass rate, user feedback). If routing to the cheap model tanks quality on a task, tighten the rule. The goal is the lowest cost at acceptable quality — a number you can only see with good logs.