Record complete function and model-attributed LLM usage
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 152
- Forks
- 16
- Avg merge
- 14h 48m
- Merged PRs (30d)
- 536
Description
Parent
Part of #610. Depends on the neutral price model from the model-pricing child issue for monetary estimates; usage attribution can be developed independently.
User outcome
Mecatl can explain which model invocations consumed a session's estimated spend, including utility calls and nested delegation, rather than exposing only coarse turn/run totals.
Problem
Current turn/result usage does not represent every LLM invocation. Utility calls such as cascade compaction can consume a provider stream without contributing usage to the session totals. Requested model identity can also differ from resolved, downstream, retried, or fallback execution.
Scope
Record exactly one durable observation for every LLM invocation made by:
- main turns;
- compaction;
- routing/classification;
- guardrails and permission review;
- learning/reflection/user-model review;
- Subagent;
- Parallel branches and judges;
- Team members and lead synthesis.
Use a small, closed, versioned functional taxonomy such as:
main
compaction
routing
safety
learning
subagent
parallel
team
Each observation should carry, where known:
- immutable call and parent identifiers;
- session/run/delegation identity for trace correlation;
- bounded function and role family;
- requested, resolved, served/downstream, retry, and fallback model/provider attribution;
- provider-reported input/output/cache-read/cache-write/reasoning usage;
- provider-reported versus locally estimated usage provenance;
- attempt, duration, outcome, stop/error category;
- estimated monetary cost, pricing coverage, price source, and price-card version.
Prefer a log-only durable usage event or equivalent observation seam. Do not make the agent loop storage-aware and do not widen port.LLMRequest with billing metadata.
Accounting rules
- One upstream attempt is observable once; aggregation must not double-count it through parent and child totals.
- Failed and retried calls retain their consumed usage and estimated cost.
- Missing usage remains unknown/estimated according to explicit provenance, never silently zero-priced.
- Served/downstream model drives pricing when trustworthy metadata exists; otherwise the estimate identifies which less-authoritative model identity it used.
- Function labels come from trusted harness structure, never prompt content or free-form model output.
Acceptance criteria
- Tests prove every in-tree LLM call site emits exactly one observation, including compaction and all delegation/safety/learning utilities.
- Nested Subagent, Parallel, and Team usage aggregates without double-counting.
- Successful, failed, retry, and fallback attempts remain distinguishable and retain usage.
- Requested/resolved/served model identities are projected honestly; absent downstream metadata stays unknown.
- Every monetary estimate carries coverage, provenance, currency/units, and price-card version.
- The event log can reconstruct one session's complete observed LLM usage after restart.
- OTel exports only bounded function/role/outcome dimensions; high-cardinality identifiers stay in traces/events.
- No prompts, raw tool content, credentials, or private route data enter financial telemetry.
- Existing token budgets retain their current semantics and remain independent of money.
- Tests are fully offline.
Out of scope
- Hard monetary enforcement.
- Free-form semantic labels such as “reading code.”
- Organization-wide analytics or chargeback.
- Automatic cost-aware routing.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inventorying every in-tree LLM call site, including compaction, delegation, safety, learning, Parallel, and Team utilities, and trace how durable state and OTel exports currently represent usage. Define the observation seam and offline tests against the acceptance criteria; done means one restart-reconstructable event per attempt, correct nested aggregation, honest model and cost provenance, and no sensitive telemetry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100