Model router proxy: local Anthropic-API-compatible LLM router with self-learning model/effort selection
- Dominant language
- Rust
- Stars
- 88
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Today, teams launch one agent with one model and every task — trivial or complex — runs through that same model regardless of actual difficulty. Proposed: a local HTTP proxy, speaking the Anthropic API protocol, that sits in front of all LLM traffic (main agent, subagents, any tool that calls out to a model) and routes each request to an appropriately-sized model based on task complexity, learning from outcomes over time.
## Architecture (from prior design session)
- **Form:** LLM proxy, not an MCP tool and not infigraph-internal-only. This is the only approach that can route *all* traffic transparently — an MCP tool can't intercept calls the agent makes directly to a model API.
- **Config mechanism:** `ANTHROPIC_BASE_URL` env var pointed at the local proxy (default port 9120). Unset the env var to disable — zero config needed to opt out.
- **Routing dimensions:** two-dimensional — model choice AND effort level (not just model swapping).
- **Cold start:** conservative. New/unseen task categories default to the strongest available model; the router only earns the right to downgrade a category to a cheaper model after it proves out over time. No manual threshold tuning expected from users.
- **Learning signal:** implicit, from observed outcomes — no explicit user feedback UI.
- Minimum 10 samples in a category before the router will downgrade it to a cheaper model.
- Retry-rate detection: if a category's retry rate (same/similar prompt reissued within a short window — a proxy for "the last response wasn't good enough") exceeds 20%, the router upgrades that category back to a stronger model.
- **Streaming:** SSE responses must pipe through the proxy transparently — this is a hard requirement, not negotiable, since agent UIs depend on streaming for responsiveness.
## Proposed components (~900 lines estimated, Rust/Axum)
1. **`infigraph-router` crate** — new crate, Axum-based HTTP server implementing the Anthropic Messages API surface.
2. **Classifier** — heuristic v1 (task-category detection from prompt/request shape); can evolve later.
3. **Model rewriter** — takes the classifier's output, looks up the current learned model+effort mapping, rewrites the outbound request.
4. **SSE streaming passthrough** — transparent relay, no buffering that would break streaming UX.
5. **Response logger** — records classification, model actually used, response time, token counts, and retry detection per request.
6. **Learning engine** — maintains task-category → model → success-rate mapping; applies the cold-start/downgrade/upgrade rules above. Needs graph storage for route decisions + outcomes (natural fit for infigraph's existing graph DB).
7. **Config** — `.infigraph/router.toml`: `enabled`, `port`, default/fallback models, per-category overrides, learning parameters (sample thresholds, retry-rate thresholds).
## Open assumptions (unvalidated, flag if wrong)
- `ANTHROPIC_BASE_URL` is assumed sufficient as the sole config mechanism — if some environments need more (e.g. per-agent routing, auth passthrough nuances), this needs revisiting.
- Retry detection (same prompt reissued in a short window) is assumed to be a reliable proxy for "response quality was insufficient." If this proves noisy in practice (e.g. legitimate retries for unrelated reasons), an explicit feedback signal may be needed instead.
## Scope
Purely additive and opt-in (env var gated) — does not change any existing infigraph behavior when unset.
Contributor guide
Research direction
Start by reviewing the repository's crate layout and existing graph-storage and configuration patterns, then scope the proposed infigraph-router crate and .infigraph/router.toml entry points. Validate the Anthropic Messages API surface, ANTHROPIC_BASE_URL behavior, SSE passthrough, and the stated learning rules. Done means an opt-in proxy routes requests, preserves streaming, logs outcomes, and learns model and effort selections without changing behavior when disabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, api, backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100