[RFC] Canonical model-call accounting: one ModelCallAttempt record per real provider request
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 715
Description
## Summary
Make one canonical record per **real provider request attempt** the source of truth for model-call accounting, and feed the Host-owned Usage authority (#1596) from it. Today the same spend is recorded by two independent writers at two different granularities, with three known gaps — and neither writer can answer "what did this turn actually cost, and is that number complete?"
This is the metering half of #1625. The Session Inspector stays downstream and becomes a pure read model over these records once they exist.
cc @likun666661 (#1649 sequencing) — see [Relationship to #1649](#relationship-to-1649).
## Problem
Two writers record model spend, neither completely:
| Path | Writer | Granularity | Carries cost? |
|---|---|---|---|
| Telemetry | `recordLlmCall` (`ai-sdk-backend.ts:1826`) | one record per **send**, aggregated across retries | yes |
| AgentRun ledger | `recordProviderRequestAttempt` (`agent-run.ts:292`) | one record per **physical attempt** | no |
Neither is authoritative, and the split produces three gaps:
1. **Provider retries carry no kind anywhere.** The main telemetry record aggregates across retries, and `provider_retry` is barred from the AgentRun ledger (`agent-run.ts:482`). A retried call is invisible as a distinct billable event.
2. **The `RuntimeEvent` ledger attributes no compaction cost.** Kernel compaction writes a phantom `input: 0, output: 0` turn (`runtime-kernel.ts:816-822`) while the real usage goes to telemetry. Anything projecting cost from the event ledger under-reports on exactly the turns that compacted.
3. **The `pi` backend records no telemetry at all.** `recordLlmCall` has zero references in `pi-agent-backend.ts`, so those sessions show tokens in the ledger and `$0` in telemetry — an indistinguishable-from-real zero.
Two more properties make the current state hard to build on:
- **Attempt writes drop silently.** Two lines apart in `runtime-kernel.ts:2462-2477`, `recordProviderRequestCapture` rejects when there's no active run while `recordProviderRequestAttempt` is `run?.recordProviderRequestAttempt(attempt)`. A late finalize after turn teardown vanishes with no trace.
- **Incompleteness is undetectable.** Nothing records an expected attempt count, and captures dedup on `(step, requestHash)`, so a partial record set can't even be recognized as partial — let alone bounded.
`callKind` covers three of the paths after #1637 (`'main' | 'semantic_compact' | 'history_compact'`, `usage-stats/types.ts:52`). Retry and `pi` remain unmodelled.
## Proposal
### One canonical record
Extend the existing per-attempt shape (`ProviderRequestAttemptRecord`, `provider-request-telemetry.ts:53`) into the canonical accounting record. It already carries identity, timing, and tokens; what it lacks is kind, cost, and honesty about coverage.
```ts
export interface ModelCallAttempt {
// identity — already present
traceId: string; // groups attempts belonging to one logical call
attemptId: string;
sessionId: string;
runId: string;
turnId: string;
step: number;
attempt: number;
// what kind of call this is — extended
callKind: 'main' | 'semantic_compact' | 'history_compact';
retryOf?: string; // attemptId this retries; absent on first attempt
// provider identity — already present
providerId: string;
modelId: string;
contextWindow?: number;
// timing — already present
startedAt: number;
completedAt: number;
latencyMs: number;
timeToFirstTokenMs?: number;
// outcome — already present
status: 'completed' | 'failed' | 'interrupted' | 'aborted';
finishReason?: string;
errorClass?: string;
// tokens — already present via ProviderRequestUsage
inputTokens?: number;
outputTokens?: number;
cacheReadInputTokens?: number;
cacheMissInputTokens?: number;
cacheWriteInputTokens?: number;
reasoningTokens?: number;
// accounting — new, and deliberately honest
costUsd?: number; // absent = unknown, never coerced to 0
costBasis?: 'priced' | 'unpriced' | 'aggregate';
granularity: 'attempt' | 'aggregate'; // 'aggregate' = backend can't split (pi)
}
```
Two fields carry the honesty requirement:
- **`granularity: 'aggregate'`** marks a record that stands for an unknown number of physical attempts. `pi` produces these. A consumer may sum them for totals but must never present them as per-attempt truth.
- **`costBasis: 'unpriced'`** marks a real call whose cost could not be resolved. Distinct from `costUsd: 0`, which must only ever mean genuinely free.
Retries stop being invisible: each physical attempt is its own record, linked by `retryOf` and grouped by `traceId`.
### Where it lives
As a new `AgentRunEvent` type, extending the seam `provider_request_attempt_recorded` already occupies (`AGENT_RUN_EVENT_TYPES`, `core/agent-run.ts:168`). No new table, no new store, no new writer.
This is deliberate: per @Astro-Han's note on #1625, an `AgentRunEvent` type needs no coordination with #1649, because that migration carries whatever event types exist and keeps store APIs stable across the cutover.
### Collection seam
`ProviderRequestTracker` (`provider-request-telemetry.ts:157`) already sits at the real `doStream` / `doGenerate` boundary and observes step, attempt, latency, and tokens. Three changes make it canonical rather than diagnostic:
1. **Fail loudly.** `recordProviderRequestAttempt` gets the same treatment as `recordProviderRequestCapture` — a missing active run is an error, not a silent return.
2. **Unbind from request capture.** Attempt recording currently rides the capture path (`provider-request-telemetry.ts:261`); accounting must not be conditional on diagnostic capture being enabled.
3. **One path for every kind.** `main`, retries, `semantic_compact`, and `history_compact` all emit through this seam, so no billable call reaches a ledger by a private route.
### Usage authority and coverage
Records feed the Host Usage/Pricing authority (#1596) — extending the existing single writer, not adding a second one. Rollups gain explicit coverage rather than exposing a single confident number:
```ts
interface UsageCoverage {
pricedCalls: number;
unpricedCalls: number; // real spend, cost unknown
aggregateCalls: number; // pi-style, attempt count unknown
}
```
A `totalCostUsd` presented without its coverage is the failure mode this whole change exists to remove.
### Removing the double write
Once the canonical record lands, the send-level `recordLlmCall` and the AgentRun attempt-usage write are redundant. Both get removed in favour of projecting telemetry from the canonical records. Existing data migrates once; **no parallel old/new writing** — a dual-write period would reintroduce exactly the reconciliation problem this replaces.
## Non-goals
- **Not** turning `RuntimeEvent` into a per-attempt billing ledger. Its job stays conversation causality, recovery, and replay; it keeps per-turn aggregates plus refs to canonical records (`refs.providerRequestTraceId` already exists for this).
- **Not** touching Runtime Host ownership, election, or session admission.
- **Not** adding authority to the Desktop embedded writer path (`session-stream.ts:289`) — #1167's M5 removes it.
- **Not** the Inspector UI. That stays in #1625, downstream.
## Proposed PR breakdown
1. **Contract + tests.** `ModelCallAttempt` type, event-type registration, validation, and pure projection helpers. No behavior change.
2. **Canonical seam.** `ProviderRequestTracker` upgrade: fail-loud writes, unbind from capture, all four paths routed through it.
3. **Usage authority integration.** Records feed #1596's writer; rollups gain `UsageCoverage`; `pi` marked `aggregate`.
4. **Remove double metering.** Delete the send-level and attempt-usage writes, one-time migration.
## Open questions
1. **`pi` aggregates** — should they land as `granularity: 'aggregate'` `ModelCallAttempt` records so one query answers everything, or stay out of the canonical store with the Usage layer merging two sources? I lean toward the former for a single read path.
2. **Cost resolution point** — resolve `costUsd` at record time (frozen at call time, matching what `RuntimeEvent` already stores) or at projection time from the pricing authority? I lean toward record time: a later price change shouldn't rewrite history.
3. **Migration fidelity** — old telemetry rows have no attempt granularity. Migrate them as `granularity: 'aggregate'` rather than fabricating per-attempt structure?
4. **Retry semantics** — is `retryOf` + shared `traceId` sufficient, or should there be an explicit terminal-attempt marker per logical call for "what did this call finally cost"?
Contributor guide
Research direction
Start with the proposed contract-and-tests step, then read ProviderRequestAttemptRecord and ProviderRequestTracker in provider-request-telemetry.ts, AgentRunEvent registration in core/agent-run.ts, and the referenced runtime-kernel.ts paths. Review the open questions before implementation; done means the canonical record, validation, projection helpers, and all four call paths have an agreed design and tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend-api-design, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100