intuit / intuit/infigraph

Model router proxy: local Anthropic-API-compatible LLM router with self-learning model/effort selection

Open
#31 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
88
Forks
16
PR merge metrics
No merged PRs in 30d

Description

## Summary

Today, teams launch one agent with one model and every task — trivial or complex — runs through that same model regardless of actual difficulty. Proposed: a local HTTP proxy, speaking the Anthropic API protocol, that sits in front of all LLM traffic (main agent, subagents, any tool that calls out to a model) and routes each request to an appropriately-sized model based on task complexity, learning from outcomes over time.

## Architecture (from prior design session)

- **Form:** LLM proxy, not an MCP tool and not infigraph-internal-only. This is the only approach that can route *all* traffic transparently — an MCP tool can't intercept calls the agent makes directly to a model API.
- **Config mechanism:** `ANTHROPIC_BASE_URL` env var pointed at the local proxy (default port 9120). Unset the env var to disable — zero config needed to opt out.
- **Routing dimensions:** two-dimensional — model choice AND effort level (not just model swapping).
- **Cold start:** conservative. New/unseen task categories default to the strongest available model; the router only earns the right to downgrade a category to a cheaper model after it proves out over time. No manual threshold tuning expected from users.
- **Learning signal:** implicit, from observed outcomes — no explicit user feedback UI.
- Minimum 10 samples in a category before the router will downgrade it to a cheaper model.
- Retry-rate detection: if a category's retry rate (same/similar prompt reissued within a short window — a proxy for "the last response wasn't good enough") exceeds 20%, the router upgrades that category back to a stronger model.
- **Streaming:** SSE responses must pipe through the proxy transparently — this is a hard requirement, not negotiable, since agent UIs depend on streaming for responsiveness.

## Proposed components (~900 lines estimated, Rust/Axum)

1. **`infigraph-router` crate** — new crate, Axum-based HTTP server implementing the Anthropic Messages API surface.
2. **Classifier** — heuristic v1 (task-category detection from prompt/request shape); can evolve later.
3. **Model rewriter** — takes the classifier's output, looks up the current learned model+effort mapping, rewrites the outbound request.
4. **SSE streaming passthrough** — transparent relay, no buffering that would break streaming UX.
5. **Response logger** — records classification, model actually used, response time, token counts, and retry detection per request.
6. **Learning engine** — maintains task-category → model → success-rate mapping; applies the cold-start/downgrade/upgrade rules above. Needs graph storage for route decisions + outcomes (natural fit for infigraph's existing graph DB).
7. **Config** — `.infigraph/router.toml`: `enabled`, `port`, default/fallback models, per-category overrides, learning parameters (sample thresholds, retry-rate thresholds).

## Open assumptions (unvalidated, flag if wrong)

- `ANTHROPIC_BASE_URL` is assumed sufficient as the sole config mechanism — if some environments need more (e.g. per-agent routing, auth passthrough nuances), this needs revisiting.
- Retry detection (same prompt reissued in a short window) is assumed to be a reliable proxy for "response quality was insufficient." If this proves noisy in practice (e.g. legitimate retries for unrelated reasons), an explicit feedback signal may be needed instead.

## Scope

Purely additive and opt-in (env var gated) — does not change any existing infigraph behavior when unset.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the repository's crate layout and existing graph-storage and configuration patterns, then scope the proposed infigraph-router crate and .infigraph/router.toml entry points. Validate the Anthropic Messages API surface, ANTHROPIC_BASE_URL behavior, SSE passthrough, and the stated learning rules. Done means an opt-in proxy routes requests, preserves streaming, logs outcomes, and learns model and effort selections without changing behavior when disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, api, backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.