stacklok / stacklok/mecatl

Record complete function and model-attributed LLM usage

Open
#612 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

Parent

Part of #610. Depends on the neutral price model from the model-pricing child issue for monetary estimates; usage attribution can be developed independently.

User outcome

Mecatl can explain which model invocations consumed a session's estimated spend, including utility calls and nested delegation, rather than exposing only coarse turn/run totals.

Problem

Current turn/result usage does not represent every LLM invocation. Utility calls such as cascade compaction can consume a provider stream without contributing usage to the session totals. Requested model identity can also differ from resolved, downstream, retried, or fallback execution.

Scope

Record exactly one durable observation for every LLM invocation made by:

  • main turns;
  • compaction;
  • routing/classification;
  • guardrails and permission review;
  • learning/reflection/user-model review;
  • Subagent;
  • Parallel branches and judges;
  • Team members and lead synthesis.

Use a small, closed, versioned functional taxonomy such as:

main
compaction
routing
safety
learning
subagent
parallel
team

Each observation should carry, where known:

  • immutable call and parent identifiers;
  • session/run/delegation identity for trace correlation;
  • bounded function and role family;
  • requested, resolved, served/downstream, retry, and fallback model/provider attribution;
  • provider-reported input/output/cache-read/cache-write/reasoning usage;
  • provider-reported versus locally estimated usage provenance;
  • attempt, duration, outcome, stop/error category;
  • estimated monetary cost, pricing coverage, price source, and price-card version.

Prefer a log-only durable usage event or equivalent observation seam. Do not make the agent loop storage-aware and do not widen port.LLMRequest with billing metadata.

Accounting rules

  • One upstream attempt is observable once; aggregation must not double-count it through parent and child totals.
  • Failed and retried calls retain their consumed usage and estimated cost.
  • Missing usage remains unknown/estimated according to explicit provenance, never silently zero-priced.
  • Served/downstream model drives pricing when trustworthy metadata exists; otherwise the estimate identifies which less-authoritative model identity it used.
  • Function labels come from trusted harness structure, never prompt content or free-form model output.

Acceptance criteria

  • Tests prove every in-tree LLM call site emits exactly one observation, including compaction and all delegation/safety/learning utilities.
  • Nested Subagent, Parallel, and Team usage aggregates without double-counting.
  • Successful, failed, retry, and fallback attempts remain distinguishable and retain usage.
  • Requested/resolved/served model identities are projected honestly; absent downstream metadata stays unknown.
  • Every monetary estimate carries coverage, provenance, currency/units, and price-card version.
  • The event log can reconstruct one session's complete observed LLM usage after restart.
  • OTel exports only bounded function/role/outcome dimensions; high-cardinality identifiers stay in traces/events.
  • No prompts, raw tool content, credentials, or private route data enter financial telemetry.
  • Existing token budgets retain their current semantics and remain independent of money.
  • Tests are fully offline.

Out of scope

  • Hard monetary enforcement.
  • Free-form semantic labels such as “reading code.”
  • Organization-wide analytics or chargeback.
  • Automatic cost-aware routing.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inventorying every in-tree LLM call site, including compaction, delegation, safety, learning, Parallel, and Team utilities, and trace how durable state and OTel exports currently represent usage. Define the observation seam and offline tests against the acceptance criteria; done means one restart-reconstructable event per attempt, correct nested aggregation, honest model and cost provenance, and no sensitive telemetry.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.