LLM model routing: what it is and when it actually saves money
Cheap model for easy calls, expensive model for hard ones, and a way to prove it did not hurt.
Quick answer
Model routing picks a model per request: short, simple prompts go to a cheaper model, and long, hard or structured ones stay on the model you asked for. It saves money when a real share of your traffic is easy and the price gap is large. It costs you when it routes tool calls or strict JSON, or when wrong answers cause retries. Route conservatively and log every decision.
Where the savings come from
The gap between tiers is large. On OpenAI's pricing page, GPT-5.4 is $2.50 per million input tokens and $15 per million output; GPT-5.4 mini is $0.75 and $4.50. Claude Sonnet 4.6 to Haiku 4.5 is a similar 3x. If 40% of your calls are "summarize this", "classify that" or "rewrite in one sentence", routing those cuts the bill for that 40% by about two thirds.
If every call is a long reasoning task, there is nothing to route, and a router just adds a decision you have to trust.
Three ways to route
- Rules. Route by task shape: prompt length, number of turns, words like "summarize" or "extract". Cheap, predictable, easy to debug. Start here.
- A classifier. A small model scores difficulty and picks the tier. Better coverage, but it is another call to pay for and another thing to be wrong.
- Cascades. Try the cheap model first and escalate when the answer fails a check. Good for structured output you can validate; slow when it escalates often.
def pick_model(messages, tools=None, response_format=None):
requested = "gpt-5.4"
prompt = messages[-1]["content"]
# Never route what can break: tools, strict JSON schemas, reasoning models.
if tools or response_format:
return requested
short = sum(len(m["content"]) for m in messages) < 3500 * 4
simple = any(w in prompt.lower() for w in ("summarize", "classify", "extract", "translate"))
return "gpt-5.4-mini" if short and simple else requestedWhat not to route
- Tool calls. Smaller models pick the wrong tool or malform arguments, and agents then loop. Keep tool-using calls on the requested model.
- Strict JSON schemas. A cheaper model that breaks the schema costs a retry, which wipes out the saving.
- Reasoning models. If someone chose an o-series or a thinking model, they chose it for a reason.
- Free models. There is nothing to save, so leave them alone.
Measure it, or it did not happen
Log the requested model, the served model and the reason for every routed call, then compare error rates, retries and user feedback on routed versus unrouted traffic for the same task. If the cheaper model fails, fall back to the requested one on the same request rather than returning an error. And re-check your targets: providers retire models, and a router pointing at a retired model either errors or silently falls back to the expensive one.
Doing this with Prune
Prune routes inside the gateway, so your code keeps asking for the model it wants. In the dashboard's Auto mode, routing is on for calls without a JSON schema (balanced, or aggressive for very short simple prompts). Per call, X-Prune-Model-Policy sets off, strict, balanced or aggressive. Balanced never routes tool calls, schema-bound calls or reasoning models, and free models are never routed at all.
Every response says what happened in headers: X-Prune-Route-Applied, X-Prune-Original-Model, X-Prune-Routed-Model and X-Prune-Route-Reason. If the routed model errors, Prune retries the model you asked for and sets X-Prune-Route-Fallback: true. The saving shows up on the signed receipt for that call.
raw = client.chat.completions.with_raw_response.create(
model="gpt-5.4",
messages=[{"role": "user", "content": "Summarize this ticket in one sentence: ..."}],
extra_headers={"X-Prune-Model-Policy": "balanced"}, # off | strict | balanced | aggressive
)
resp = raw.parse()
raw.headers["X-Prune-Route-Applied"] # "true"
raw.headers["X-Prune-Original-Model"] # gpt-5.4
raw.headers["X-Prune-Routed-Model"] # gpt-5.4-mini
raw.headers["X-Prune-Route-Reason"] # simple_short_promptWith a single line of code, Prune cuts your AI bill and protects your team from runaway spend, leaked keys and dangerous calls.
