Multi-provider failover
route-switch fronts six providers behind one OpenAI-compatible endpoint and falls back to the next eligible combination when a provider's success rate drops below a threshold — turning provider switching from a deploy into a policy.
A single provider is a single point of failure. When OpenAI has a bad hour, an application wired directly to its SDK has a bad hour too. route-switch makes provider redundancy a configuration property instead of a code path.
The shape
Register the same prompt against multiple providers as separate
combinations, mark resilient ones as fallback: true, and let the load balancer route.
When a combination’s recent success rate drops below its threshold, the balancer moves to the next
eligible combination — see choosing a routing strategy.
combinations:
- prompt: support-reply
provider: openai
model: gpt-4
weight: 3
- prompt: support-reply
provider: anthropic
model: claude-3-5-sonnet
weight: 1
fallback: trueWhy a gateway, not an if-tree
A hand-rolled if (openaiDown) branch welds fallback logic into every service, scatters provider
credentials, and requires a deploy to change. The gateway lifts all three: fallback is a policy,
credentials live in one place, and switching providers is a config edit. Pair it with
trace capture so you can see which combination actually carried
traffic during an incident.
route-switch is self-hosted and MIT-licensed, built by Skelf Research.
Frequently asked questions
How does failover trigger?
Each combination has a success-rate threshold. When a combination's recent success drops below it, the load balancer redirects to the next eligible combination automatically — no deploy, no code change.
Do I need accounts with every provider?
Only the ones you route to. route-switch is bring-your-own-key: you supply credentials per provider in config.yaml for the providers you actually use.
Related
Latency-aware routing
Use performance-based routing and DuckDB latency data to favor combinations that respond faster, with fallback when a provider slows down.
Use caseCost-aware routing
Bias traffic toward cheaper model combinations with weighted routing, and see per-prompt cost in DuckDB so routing decisions are grounded in real spend.
Use casePrompt optimization from production traces
Close the loop: capture real traffic, replay it to score MIPROv2 candidates, and ship the winning prompt — automatically, on an interval.