Cost-aware routing
route-switch uses weighted routing to bias traffic toward cheaper combinations and records per-call cost in DuckDB, so you can route on real spend per prompt rather than guesswork.
Most LLM cost surprises come from not knowing which prompt is expensive. route-switch records cost per call and lets you bias routing toward cheaper combinations — so spend is a dial, not a monthly shock.
Measure first
Every served call writes its cost into the per-prompt SQLite dataset and the DuckDB analytics store. Query cost per prompt directly, or read the rollup:
curl http://localhost:8080/v1/system/analyticsSee capturing traces to DuckDB for the full schema.
Then bias the routing
With grounded per-prompt cost, use weighted routing to send most traffic to a cheaper combination and a slice to a stronger one:
gateway:
strategy: weighted
combinations:
- prompt: classify
provider: openai
model: gpt-4o-mini
weight: 9
- prompt: classify
provider: openai
model: gpt-4
weight: 1The honest limit
route-switch does not invent a price for you or claim a percentage saving — it gives you the measurement and the routing lever, and the trade-off between cost and quality stays a decision you make with your own data. Combine with latency-aware routing when speed also matters.
Frequently asked questions
Does route-switch decide cost automatically?
It gives you the levers: weighted routing to bias toward cheaper combinations, and per-call cost recorded in DuckDB so you can measure the trade-off. The policy is yours to set.
Where does the cost figure come from?
Each logged call records its cost in the per-prompt SQLite dataset and DuckDB store, aggregated per prompt and globally through the analytics endpoints.
Related
Latency-aware routing
Use performance-based routing and DuckDB latency data to favor combinations that respond faster, with fallback when a provider slows down.
Use caseMulti-provider failover
Front six providers behind one endpoint, and fall back automatically when a provider combination's success rate drops below its threshold.
Use casePrompt optimization from production traces
Close the loop: capture real traffic, replay it to score MIPROv2 candidates, and ship the winning prompt — automatically, on an interval.