terraphim / terraphim/terraphim-ai
Budget-aware multi-model routing with per-task cost tracking
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 62
- Forks
- 5
- Avg merge
- 2h 27m
- Merged PRs (30d)
- 1
Description
Summary
Add per-task cost tracking with fallback chains and hard spending limits to LLM routing. Each agent operation gets a budget allocation; if the primary model is too expensive or unavailable, fall back to cheaper alternatives automatically.
Motivation
Inspired by Ouroboros per-round cost tracking with per-worker budget allocation. Current state:
terraphim-llm-proxyroutes between multiple models but has no per-task budget trackinggenai(terraphim fork) handles LLM calls but does not meter usage or cost- The multi-model judge pattern (quick -> deep -> tiebreaker) needs budget awareness: only escalate to expensive models when cheaper ones disagree
- No spending limits exist -- a runaway agent loop could consume unbounded API credits
Implementation Plan
1. Budget Struct
pub struct TaskBudget {
pub max_cost_usd: f64,
pub model_chain: Vec<ModelConfig>, // ordered by preference (cheapest first)
pub spent_usd: f64,
pub request_count: u32,
pub created_at: jiff::Timestamp,
}
pub struct ModelConfig {
pub model_id: String,
pub cost_per_1k_input: f64,
pub cost_per_1k_output: f64,
pub max_latency_ms: u64,
pub fallback_on: Vec<FallbackReason>, // Timeout, RateLimit, BudgetExceeded, EmptyResponse
}
2. Cost Tracking in terraphim_service
- After each LLM call, compute cost from token counts + model pricing
- Accumulate in
TaskBudget.spent_usd - If
spent_usd >= max_cost_usd, reject further calls for this task - Log cost events to event store (#597)
3. Fallback Chain Logic
- Try models in
model_chainorder - On failure (timeout, rate limit, empty response, budget exceeded for this model tier), try next model
- If all models exhausted, return error with cost summary
4. Integration with terraphim-llm-proxy
- Proxy already reports token counts in responses
- Add cost metadata to proxy response headers (cost_usd, model_used, tokens_in, tokens_out)
- Agent-side budget tracking consumes these headers
5. MCP Resource Exposure
- Expose
/budget/currentas MCP resource showing active task budgets - Expose
/budget/historyfor cost analysis over time
Affected Crates
terraphim_service(primary -- add budget tracking to LLM routing)terraphim_types(addTaskBudget,ModelConfigtypes)terraphim_multi_agent(wire budget into agent task execution)terraphim_mcp_server(expose budget resources)
Dependencies
- #597 Event sourcing (for cost event persistence)
Estimated Effort
~4 hours (mostly plumbing between terraphim_service and proxy response headers)
Part of
Epic #595
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the affected terraphim_service, terraphim_types, terraphim_multi_agent, and terraphim_mcp_server crates, then review the event-sourcing dependency in #597 and the existing terraphim-llm-proxy token headers. Done means task budgets meter costs, enforce limits, fall back through model chains, persist cost events, and expose the current and historical MCP budget resources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100