How route-switch works
One Go process holds the prompt registry, the router, the provider adapter, and the analytics store. A request flows through seven stages; the optimizer closes the loop back to the registry.
OpenAI client route-switch (single Go binary)
┌───────────┐ POST /v1/chat/completions ┌───────────────────────────────────────────┐
│ your app │ ─────────────────────────────▶│ ┌─────────────┐ ┌──────────────────────┐│
└───────────┘ │ │ prompt │ │ load balancer ││
│ │ registry │──▶│ round_robin / ││
rendered prompt │ │ (YAML) │ │ weighted / ││
│ └─────────────┘ │ performance_based ││
│ └──────────┬───────────┘│
│ │ gollm │
│ ┌───────────────┐ ▼ │
│ │ DuckDB + per- │◀── log ── provider call ─┼──▶ OpenAI · Anthropic
│ │ prompt SQLite │ │ │ Google · Ollama
│ └──────┬────────┘ │ │ Cohere · Mistral
│ │ replay │ │
│ ┌──────▼────────┐ │ │
│ │ MIPROv2 │── writes ─┘ (new prompt)│
│ │ optimizer │──────────────────────────┘
│ └───────────────┘ (goptuna Bayesian search)
└───────────────────────────────────────────┘Request in
An OpenAI-compatible client POSTs to /v1/chat/completions with a model, messages, and optional variables + template metadata.
Template render
The gateway resolves the registered prompt template by ID, validates variables against its schema, and renders the final prompt server-side.
Route
The load balancer picks one registered (prompt, model, provider) combination by strategy: round_robin, weighted, or performance_based.
Provider call
The chosen combination is dispatched through the gollm adapter to OpenAI, Anthropic, Google, Ollama, Cohere, or Mistral, using your credentials.
Fallback
If the combination's recent success rate is below the configured threshold, the balancer falls back to the next eligible combination.
Log
The call is written to the per-prompt SQLite dataset (rendered input, output, variables, model, provider, success, cost) and to the DuckDB analytics store.
Optimize
On its interval, the MIPROv2 optimizer replays dataset rows, scores instruction + few-shot candidates via goptuna, and writes the winner back to the registry.
State the gateway keeps
- • Per-prompt SQLite datasets (one file per template) under
dataset.base_path, trimmed todataset.max_records. - • DuckDB analytics store: request/response summaries, latency, error rates, and cost rollups.
- • YAML template manifests, one per registered prompt.