Routing is the practice of sending each request to the cheapest model that can handle it, and a cascade is its most common form: try a small model, check the answer, and escalate only if the check fails. The idea is simple and the prices make it tempting, but the savings reported in papers do not transfer automatically to your workload. This guide explains the two main designs, works through the arithmetic on Anthropic's current price list, and sets out the conditions under which routing is worth building.
Prices and API rules below were read from Anthropic's documentation on 2026-10-12. The research claims come from the abstracts of two arXiv papers, which we opened on the same date; we did not reproduce their experiments, and we did not read their full results sections. Anything labelled "worked example" is our own arithmetic on stated assumptions, not a measurement.
Quick answer
- Router: decide first, call one model. A rule, a classifier or a trained model picks the model before any answer exists.
- Cascade: call the cheap model first, check the answer, and escalate on failure. You always pay the cheap model, and you pay the expensive one only on the escalated fraction.
- Why it can work: FrugalGPT notes that fees across LLM APIs can differ by two orders of magnitude, and Anthropic's own tiers span a similar range.[1][3]
- Why it often does not: the check that decides to escalate costs money and makes mistakes, and every wrong "good enough" is a quality failure you may not see.
- Measure first: build an evaluation set, run the cheap model on it, and find out what fraction it gets right. See How to Build LLM Evals for Your Application.
The price spread that makes routing worth discussing
Anthropic's pricing page lists per-million-token prices for each model. These are the current rows relevant to a routing decision, with Haiku 5.5 shown at its lower price band, which applies to prompts up to 100,000 tokens.[3]
| Model | Input per MTok | Output per MTok | Cache read per MTok |
|---|---|---|---|
| Claude Haiku 5.5 (prompts up to 100,000 tokens) | $0.10 | $0.50 | $0.01 |
| Claude Sonnet 5.5 | $2 | $10 | $0.10 |
| Claude Opus 5.5 | $4 | $20 | $0.20 |
| Claude Fable 5.1 | $10 | $50 | $0.25 |
Source: Anthropic pricing page, read 2026-10-12.[3] Haiku 5.5 is priced higher for prompts over 100,000 tokens ($0.50 input, $2.50 output), which narrows the gap on long-context work; see Claude Haiku 5.5: Specs, Pricing, Migration Traps and When to Use It for that break.
Arithmetically, Sonnet 5.5 costs 20 times Haiku 5.5 on both input and output at the lower band, and Fable 5.1 costs 100 times Haiku 5.5. That ratio is why a router has room to pay for itself. It also shows the limit: a price ratio is only an opportunity. Whether the cheap model is adequate is a quality question that the price page cannot answer.
Anthropic's own cost guidance on the same page is a coarse version of routing: use Haiku for simple tasks, Sonnet for most production workloads and Opus for the most complex reasoning.[3] That is a heuristic by task type, not a measured policy.
Two designs
Routers decide before answering
A router receives the request and picks a model. The decision can be a hand-written rule (short classification prompts go to the small model, anything with a long document goes to the larger one), a classifier, or a learned model. RouteLLM is the learned version: its authors propose router models that dynamically select between a stronger and a weaker LLM during inference, trained using human preference data, and report that this reduces cost "by over 2 times in certain cases" without compromising response quality on the benchmarks they evaluated. They also report that the routers kept their performance when the strong and weak models were changed at test time.[2]
The abstract says the evaluation used "widely-recognized benchmarks" and does not name them, so we cannot say here which tasks the result covers. Treat "over 2x in certain cases" as an existence proof, not a forecast.
Cascades answer first and check later
A cascade pays for a draft. FrugalGPT describes three families of cost reduction: prompt adaptation, LLM approximation and LLM cascades. It proposes a cascade that learns which models to use for different queries, and reports that it can match the best individual model in its study (GPT-4) with up to 98% cost reduction, or improve accuracy over GPT-4 by 4% at the same cost.[1]
Two cautions apply. The paper was submitted in May 2023, so its model set and price ratios are not today's. And "up to" is a best case across the datasets the authors chose. Both reported numbers are still useful because they show the two ways a cascade can win: by saving money at equal quality, or by improving quality at equal cost, since a cheaper model is sometimes right where the expensive one is wrong.
A worked cost model
Assume a hypothetical workload of one million requests a month, each with 2,000 input tokens and 300 output tokens. This is an assumption chosen for easy arithmetic, not a typical size, and it holds token counts equal across models, which ignores the tokenizer differences discussed in Tokens, Context Windows and What an AI Request Really Costs.
Cost per request is input tokens times the input price plus output tokens times the output price:
| Model | Per request | Per million requests |
|---|---|---|
| Haiku 5.5 | $0.00035 | $350 |
| Sonnet 5.5 | $0.007 | $7,000 |
| Opus 5.5 | $0.014 | $14,000 |
| Fable 5.1 | $0.035 | $35,000 |
Now build a Haiku-first cascade that escalates a fraction f of requests to Sonnet 5.5. Every request pays Haiku, and the escalated ones also pay Sonnet again for the full request:
cascade cost = 350 + f x 7,000 (dollars per million requests)| Escalation rate f | Cascade cost | Versus Sonnet only |
|---|---|---|
| 10% | $1,050 | 85% cheaper |
| 30% | $2,450 | 65% cheaper |
| 60% | $4,550 | 35% cheaper |
| 95% | $7,000 | break-even |
The cascade stays cheaper than going straight to Sonnet until f reaches 95%, because the Haiku pass is only 5% of the Sonnet price. That is a property of this price pair, not a general result. Between Sonnet and Fable the ratio is only 5x, so the first pass is a larger share and the break-even escalation rate is lower.
What this table leaves out matters more than what it contains:
- The check is not free. Deciding whether the Haiku answer passes needs a rule, a schema validator, or another model call. A validator that checks JSON against a schema is almost free. An LLM judge adds a call to every request and has its own error rate.
- False passes are invisible. If the check lets a wrong cheap answer through, the table shows savings while quality quietly drops. If the check escalates good answers, you pay twice for nothing.
- Latency adds up. An escalated request waits for the cheap model, the check and then the stronger model.
- Caching and long prompts change the numbers. A cached prefix is read at a fraction of the input price, and Haiku 5.5 prompts over 100,000 tokens move to the higher band.[3]
When routing is worth it
Routing tends to pay when all of these hold:
- Volume is high enough that engineering time is small next to the monthly bill. At $350 versus $7,000 per million requests the gap is visible; at a few thousand requests it is not.
- Requests are heterogeneous. If easy and hard cases are mixed, there is something to sort. If every request is equally hard, a router has nothing to exploit.
- Correctness can be checked cheaply. Structured extraction, classification into a fixed set, code that must pass tests and answers that must match a schema all give a cheap pass or fail. Open-ended writing does not.
- You have an evaluation set. Without one you cannot measure the cheap model's accuracy or the check's false-pass rate. Why AI Models Hallucinate, and What Actually Reduces It explains why confident wrong answers make this hard.
It is probably not worth it when volume is small, when a wrong answer is expensive, or when the cheap model already handles nearly everything: then use it alone and skip the machinery.
Levers that are not routing
Some savings need no quality trade-off, so try them first.
- Batching. Anthropic's pricing page lists a 50% discount on both input and output tokens for the Batch API.[3] The batch documentation says most batches finish within an hour, results are available when all requests complete or after 24 hours, whichever is first, and batches expire if not finished within 24 hours. A batch is limited to 100,000 requests or 256 MB.[4] It suits evaluation runs and bulk processing, not interactive use. Fast mode is not available with batches.[3][4]
- Prompt caching. Cache reads are priced at a fraction of base input, from 0.025x to 0.1x depending on the model.[3]
- Shorter outputs. Output tokens are priced at 5x input on every row in the table above, so trimming verbose answers saves more per token than trimming prompts.[3]
A practical adoption path
- Log real traffic and sample it into an evaluation set with correct answers or a scoring rule.
- Run the cheap model alone on that set and record accuracy per task type.
- Route by rule first. Send the task types where the cheap model is clearly adequate to it and the rest to the stronger model. This needs no learned router.
- Add a cheap check only where one exists, and measure its false-pass rate on the evaluation set before trusting it.
- Recompute when models change. A new release changes both the quality and the price of each tier, so rerun the evaluation instead of assuming last quarter's routing table still holds.
What remains uncertain
We have not tested a router or cascade ourselves, so every savings figure here is either a paper's own reported result or arithmetic on stated assumptions. The papers' abstracts do not tell us the datasets behind RouteLLM's headline figure, and FrugalGPT's models are from 2023. Prices change; check the pricing page before budgeting.
