terraphim / terraphim/terraphim-ai

Budget-aware multi-model routing with per-task cost tracking

Open
#598 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

architecture enhancement
Dominant language
Rust
Stars
62
Forks
5
Avg merge
2h 27m
Merged PRs (30d)
1

Description

Summary

Add per-task cost tracking with fallback chains and hard spending limits to LLM routing. Each agent operation gets a budget allocation; if the primary model is too expensive or unavailable, fall back to cheaper alternatives automatically.

Motivation

Inspired by Ouroboros per-round cost tracking with per-worker budget allocation. Current state:

  • terraphim-llm-proxy routes between multiple models but has no per-task budget tracking
  • genai (terraphim fork) handles LLM calls but does not meter usage or cost
  • The multi-model judge pattern (quick -> deep -> tiebreaker) needs budget awareness: only escalate to expensive models when cheaper ones disagree
  • No spending limits exist -- a runaway agent loop could consume unbounded API credits

Implementation Plan

1. Budget Struct
pub struct TaskBudget {
    pub max_cost_usd: f64,
    pub model_chain: Vec<ModelConfig>,  // ordered by preference (cheapest first)
    pub spent_usd: f64,
    pub request_count: u32,
    pub created_at: jiff::Timestamp,
}

pub struct ModelConfig {
    pub model_id: String,
    pub cost_per_1k_input: f64,
    pub cost_per_1k_output: f64,
    pub max_latency_ms: u64,
    pub fallback_on: Vec<FallbackReason>,  // Timeout, RateLimit, BudgetExceeded, EmptyResponse
}
2. Cost Tracking in terraphim_service
  • After each LLM call, compute cost from token counts + model pricing
  • Accumulate in TaskBudget.spent_usd
  • If spent_usd >= max_cost_usd, reject further calls for this task
  • Log cost events to event store (#597)
3. Fallback Chain Logic
  • Try models in model_chain order
  • On failure (timeout, rate limit, empty response, budget exceeded for this model tier), try next model
  • If all models exhausted, return error with cost summary
4. Integration with terraphim-llm-proxy
  • Proxy already reports token counts in responses
  • Add cost metadata to proxy response headers (cost_usd, model_used, tokens_in, tokens_out)
  • Agent-side budget tracking consumes these headers
5. MCP Resource Exposure
  • Expose /budget/current as MCP resource showing active task budgets
  • Expose /budget/history for cost analysis over time

Affected Crates

  • terraphim_service (primary -- add budget tracking to LLM routing)
  • terraphim_types (add TaskBudget, ModelConfig types)
  • terraphim_multi_agent (wire budget into agent task execution)
  • terraphim_mcp_server (expose budget resources)

Dependencies

  • #597 Event sourcing (for cost event persistence)

Estimated Effort

~4 hours (mostly plumbing between terraphim_service and proxy response headers)

Part of

Epic #595

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the affected terraphim_service, terraphim_types, terraphim_multi_agent, and terraphim_mcp_server crates, then review the event-sourcing dependency in #597 and the existing terraphim-llm-proxy token headers. Done means task budgets meter costs, enforce limits, fall back through model chains, persist cost events, and expose the current and historical MCP budget resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.