Glossary
The vocabulary of route-switch and LLM gateways, defined precisely.
- LLM gateway
- A service that sits between your application and one or more model providers, presenting a single API surface and centralizing routing, prompt management, and analytics. route-switch is a self-hosted LLM gateway.
- OpenAI-compatible endpoint
- An HTTP endpoint that accepts the same request/response shape as OpenAI's /v1/chat/completions, so existing OpenAI clients work by changing only the base URL.
- Combination
- A registered tuple of (prompt template, model, provider, weight, fallbacks). The load balancer routes each request across the combinations you have registered, not across arbitrary models.
- Routing strategy
- The rule the load balancer uses to pick a combination: round_robin (even distribution), weighted (biased by configured weight), or performance_based (favoring better observed success).
- Fallback
- A policy that redirects a request to the next eligible combination when the chosen combination's recent success rate drops below a configured threshold.
- Prompt registry
- The gateway's store of prompt templates as YAML manifests with variable schemas, addressable by ID. The gateway owns the prompt, rather than it living in the request payload.
- MIPROv2
- A prompt-optimization algorithm that performs a joint instruction and few-shot demonstration search. route-switch drives it with goptuna's Bayesian optimizer against captured datasets.
- goptuna
- A Go Bayesian optimization library. route-switch uses it to search the space of instruction and few-shot demo combinations during MIPROv2 optimization.
- gollm
- The Go LLM provider adapter route-switch uses to fan a single request out to OpenAI, Anthropic, Google, Ollama, Cohere, Mistral, and others.
- DuckDB
- An in-process analytical database. route-switch writes per-call summaries — success, latency, cost — to a local DuckDB store for per-prompt and global aggregates.
- Evaluation strategy
- The scorer the optimizer applies when replaying dataset rows: Similarity (token overlap), ExactMatch (trimmed equality), or KeywordMatch (declared keyword presence) — or a custom implementation of the EvaluationStrategy Go interface.
- Trace / dataset
- The record of a served call — rendered input, output, variables, model, provider, success, cost — written to a per-prompt SQLite dataset. Traces are the raw material the optimizer replays.
- Portable package
- A tarball bundling a prompt template with its dataset snapshot and recent logs, so a prompt and its history can move between environments.