Latency-aware routing
route-switch records per-call latency in DuckDB and can favor better-performing combinations with performance_based routing, so slow combinations lose traffic and fallback covers provider slowdowns.
When you have a latency budget, routing to a slow combination is a defect. route-switch records response time per call and lets routing lean on combinations that perform.
Latency is in the data
Every call’s latency lands in the DuckDB analytics store alongside success and cost. Read it per prompt or globally — see capturing traces to DuckDB — so you know which combination is actually fast under your traffic, not in a vendor benchmark.
Route on observed performance
performance_based routing favors combinations with better
observed outcomes. Combined with per-combination fallback thresholds, a combination that degrades
loses traffic and, if it drops far enough, is skipped entirely:
gateway:
strategy: performance_based
combinations:
- prompt: draft-email
provider: openai
model: gpt-4o
fallback: true
- prompt: draft-email
provider: mistral
model: mistral-large
fallback: trueWhat it is not
route-switch does not promise a latency SLA or publish speed benchmarks — it measures your real latency and gives you routing levers to act on it. For the cost side of the same trade-off, see cost-aware routing.
Frequently asked questions
Does performance_based routing consider latency?
It favors combinations with better observed performance from captured traffic. Latency is recorded per call in DuckDB, so you can read it directly and design your combinations and thresholds around it.
What happens when a provider gets slow?
If slowness pushes a combination's success rate below its fallback threshold, the balancer redirects to the next eligible combination. You can also read latency trends and adjust weights.
Related
Multi-provider failover
Front six providers behind one endpoint, and fall back automatically when a provider combination's success rate drops below its threshold.
Use caseCost-aware routing
Bias traffic toward cheaper model combinations with weighted routing, and see per-prompt cost in DuckDB so routing decisions are grounded in real spend.
Use casePrompt optimization from production traces
Close the loop: capture real traffic, replay it to score MIPROv2 candidates, and ship the winning prompt — automatically, on an interval.