Running MIPROv2 on production traces

route-switch implements MIPROv2 — a joint search over instructions and few-shot demonstrations — and runs it against the traffic the gateway already captured. This is the closed loop that distinguishes route-switch from a plain routing layer.

The loop, concretely

1. Bootstrap a calibration sample from the prompt’s SQLite dataset.
2. Generate instruction candidates by calling the configured provider with an instruction-proposal meta-prompt.
3. Search — drive a Bayesian optimization (goptuna) over (instruction, few-shot demo) combinations.
4. Score each candidate by replaying dataset rows under an evaluation strategy: Similarity, ExactMatch, KeywordMatch, or a custom Go implementation.
5. Write the winning prompt back to the registry.

Running it

gateway:
  optimization:
    enabled: true
    interval: 6h     # background cadence

Or trigger a one-off run from the CLI with --optimize-prompt. Either way, optimization is per-prompt and per-(prompt, model) — it does not touch other templates.

Honesty about the loss function

The optimizer scores against your captured traffic with the strategy you configured. If Similarity (token overlap), ExactMatch (trimmed equality), and KeywordMatch (declared-keyword presence) don’t capture what “good” means for your task, implement the EvaluationStrategy interface — otherwise you’ll optimize toward the wrong target.

It reliably moves format adherence, demo selection, and instruction specificity. It will not fix a task the model can’t do, a mismatched scorer, or a non-representative dataset. Set up trace capture first, and see prompt optimization from traces for the end-to-end story.