proposal(auto): learned cost/quality routing trained on receipts — exploration, replay labels, embedded artifact
- Dominant language
- Rust
- Stars
- 41k
- Forks
- 3.6k
- Avg merge
- 13h 59m
- Merged PRs (30d)
- 299
Description
## Context
Depends on the sticky-binding and receipts issues (#6252, #6253). This is a proposal/tracking issue, not committed scope.
Once sticky auto bindings produce receipted outcomes (binding id, model, tokens, cost, `TurnOutcomeStatus`, rebind history), the ledger becomes training data for a **learned** cost/quality router: given a session's opening request, pick the cheapest candidate likely to succeed, escalate on failure. The open-source reference points evaluated for this:
- **RouteLLM / LLMRouter** (lm-sys/RouteLLM, ulab-uiuc/LLMRouter; Apache-2.0/MIT) — trained classifiers for cost/quality routing. The practical method for our 2-candidate shape (cheap vs escalation) is LLMRouter's `racerrouter`: a ~238K-param MLP over a query embedding with an explicit cost budget. Its checkpoint is a self-describing versioned dict; the forward pass is ~50 lines of Rust, no ML runtime.
- **TensorZero** (Apache-2.0, archived 2026-06) — Rust gateway-as-library; reference for config schema and fallback-chain design, not a dependency.
## Open problems this issue tracks
1. **Counterfactual labels.** The ledger records only the candidate actually used; every published trainer needs per-candidate outcomes per prompt. Options: bounded exploration at *session* scope (a small fraction of new sessions bind to the alternate candidate — never per-turn, which would bust the prefix cache), plus offline replays of logged opening prompts against the other candidate for judge/outcome labels.
2. **Text encoder.** LLMRouter's classifiers sit on a 149M-param Longformer — too heavy to ship for a routing decision. Retrain on a small encoder exportable to ONNX (MiniLM/bge-small class) using LLMRouter's precomputed-embedding backend, keeping train-time (Python) and run-time (Rust) encoder consistency.
3. **Artifact format + Rust scoring.** Versioned checkpoint (candidate list, encoder id, weights, training summary); a tiny pure-Rust forward pass behind the same binding-creation path as scenario rules, so the learned router is one more bind-time decider with identical receipts.
4. **Evaluation harness.** Offline replay on ledger data must beat the scenario-rules baseline on cost-at-equal-quality before any live traffic sees it; exploration rate and escalation policy are config with conservative defaults.
## Non-goals
- No per-turn learned routing (cache economics forbid it — see #6252).
- No second runtime or external router service; the routing decision stays a function call inside the Engine.
- No training on customer content; embeddings and labels derive from receipted metadata under the existing telemetry boundaries.
## Acceptance (for the tracking issue itself)
A dated evaluation report: replay harness, baseline vs learned-router cost/quality on ledger-derived data, and a go/no-go recommendation for live exploration.
Contributor guide
Research direction
Start by reading the dependent sticky-binding and receipts issues (#6252 and #6253), then trace the Engine's binding-creation path and receipted ledger data. Define the replay harness, counterfactual labeling, artifact format, and Rust scoring boundary before implementation. Done means a dated baseline-versus-learned cost/quality report and a go/no-go recommendation for live exploration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- ai, backend-api-design, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100