apache / apache/maka

[RFC] Canonical model-call accounting: one ModelCallAttempt record per real provider request

Open
#1,679 6 comments 0 reactions 0 assignees View on GitHub
enhancement stale
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
715

Description

## Summary

Make one canonical record per **real provider request attempt** the source of truth for model-call accounting, and feed the Host-owned Usage authority (#1596) from it. Today the same spend is recorded by two independent writers at two different granularities, with three known gaps — and neither writer can answer "what did this turn actually cost, and is that number complete?"

This is the metering half of #1625. The Session Inspector stays downstream and becomes a pure read model over these records once they exist.

cc @likun666661 (#1649 sequencing) — see [Relationship to #1649](#relationship-to-1649).

## Problem

Two writers record model spend, neither completely:

| Path | Writer | Granularity | Carries cost? |
|---|---|---|---|
| Telemetry | `recordLlmCall` (`ai-sdk-backend.ts:1826`) | one record per **send**, aggregated across retries | yes |
| AgentRun ledger | `recordProviderRequestAttempt` (`agent-run.ts:292`) | one record per **physical attempt** | no |

Neither is authoritative, and the split produces three gaps:

1. **Provider retries carry no kind anywhere.** The main telemetry record aggregates across retries, and `provider_retry` is barred from the AgentRun ledger (`agent-run.ts:482`). A retried call is invisible as a distinct billable event.
2. **The `RuntimeEvent` ledger attributes no compaction cost.** Kernel compaction writes a phantom `input: 0, output: 0` turn (`runtime-kernel.ts:816-822`) while the real usage goes to telemetry. Anything projecting cost from the event ledger under-reports on exactly the turns that compacted.
3. **The `pi` backend records no telemetry at all.** `recordLlmCall` has zero references in `pi-agent-backend.ts`, so those sessions show tokens in the ledger and `$0` in telemetry — an indistinguishable-from-real zero.

Two more properties make the current state hard to build on:

- **Attempt writes drop silently.** Two lines apart in `runtime-kernel.ts:2462-2477`, `recordProviderRequestCapture` rejects when there's no active run while `recordProviderRequestAttempt` is `run?.recordProviderRequestAttempt(attempt)`. A late finalize after turn teardown vanishes with no trace.
- **Incompleteness is undetectable.** Nothing records an expected attempt count, and captures dedup on `(step, requestHash)`, so a partial record set can't even be recognized as partial — let alone bounded.

`callKind` covers three of the paths after #1637 (`'main' | 'semantic_compact' | 'history_compact'`, `usage-stats/types.ts:52`). Retry and `pi` remain unmodelled.

## Proposal

### One canonical record

Extend the existing per-attempt shape (`ProviderRequestAttemptRecord`, `provider-request-telemetry.ts:53`) into the canonical accounting record. It already carries identity, timing, and tokens; what it lacks is kind, cost, and honesty about coverage.

```ts
export interface ModelCallAttempt {
// identity — already present
traceId: string; // groups attempts belonging to one logical call
attemptId: string;
sessionId: string;
runId: string;
turnId: string;
step: number;
attempt: number;

// what kind of call this is — extended
callKind: 'main' | 'semantic_compact' | 'history_compact';
retryOf?: string; // attemptId this retries; absent on first attempt

// provider identity — already present
providerId: string;
modelId: string;
contextWindow?: number;

// timing — already present
startedAt: number;
completedAt: number;
latencyMs: number;
timeToFirstTokenMs?: number;

// outcome — already present
status: 'completed' | 'failed' | 'interrupted' | 'aborted';
finishReason?: string;
errorClass?: string;

// tokens — already present via ProviderRequestUsage
inputTokens?: number;
outputTokens?: number;
cacheReadInputTokens?: number;
cacheMissInputTokens?: number;
cacheWriteInputTokens?: number;
reasoningTokens?: number;

// accounting — new, and deliberately honest
costUsd?: number; // absent = unknown, never coerced to 0
costBasis?: 'priced' | 'unpriced' | 'aggregate';
granularity: 'attempt' | 'aggregate'; // 'aggregate' = backend can't split (pi)
}
```

Two fields carry the honesty requirement:

- **`granularity: 'aggregate'`** marks a record that stands for an unknown number of physical attempts. `pi` produces these. A consumer may sum them for totals but must never present them as per-attempt truth.
- **`costBasis: 'unpriced'`** marks a real call whose cost could not be resolved. Distinct from `costUsd: 0`, which must only ever mean genuinely free.

Retries stop being invisible: each physical attempt is its own record, linked by `retryOf` and grouped by `traceId`.

### Where it lives

As a new `AgentRunEvent` type, extending the seam `provider_request_attempt_recorded` already occupies (`AGENT_RUN_EVENT_TYPES`, `core/agent-run.ts:168`). No new table, no new store, no new writer.

This is deliberate: per @Astro-Han's note on #1625, an `AgentRunEvent` type needs no coordination with #1649, because that migration carries whatever event types exist and keeps store APIs stable across the cutover.

### Collection seam

`ProviderRequestTracker` (`provider-request-telemetry.ts:157`) already sits at the real `doStream` / `doGenerate` boundary and observes step, attempt, latency, and tokens. Three changes make it canonical rather than diagnostic:

1. **Fail loudly.** `recordProviderRequestAttempt` gets the same treatment as `recordProviderRequestCapture` — a missing active run is an error, not a silent return.
2. **Unbind from request capture.** Attempt recording currently rides the capture path (`provider-request-telemetry.ts:261`); accounting must not be conditional on diagnostic capture being enabled.
3. **One path for every kind.** `main`, retries, `semantic_compact`, and `history_compact` all emit through this seam, so no billable call reaches a ledger by a private route.

### Usage authority and coverage

Records feed the Host Usage/Pricing authority (#1596) — extending the existing single writer, not adding a second one. Rollups gain explicit coverage rather than exposing a single confident number:

```ts
interface UsageCoverage {
pricedCalls: number;
unpricedCalls: number; // real spend, cost unknown
aggregateCalls: number; // pi-style, attempt count unknown
}
```

A `totalCostUsd` presented without its coverage is the failure mode this whole change exists to remove.

### Removing the double write

Once the canonical record lands, the send-level `recordLlmCall` and the AgentRun attempt-usage write are redundant. Both get removed in favour of projecting telemetry from the canonical records. Existing data migrates once; **no parallel old/new writing** — a dual-write period would reintroduce exactly the reconciliation problem this replaces.

## Non-goals

- **Not** turning `RuntimeEvent` into a per-attempt billing ledger. Its job stays conversation causality, recovery, and replay; it keeps per-turn aggregates plus refs to canonical records (`refs.providerRequestTraceId` already exists for this).
- **Not** touching Runtime Host ownership, election, or session admission.
- **Not** adding authority to the Desktop embedded writer path (`session-stream.ts:289`) — #1167's M5 removes it.
- **Not** the Inspector UI. That stays in #1625, downstream.

## Proposed PR breakdown

1. **Contract + tests.** `ModelCallAttempt` type, event-type registration, validation, and pure projection helpers. No behavior change.
2. **Canonical seam.** `ProviderRequestTracker` upgrade: fail-loud writes, unbind from capture, all four paths routed through it.
3. **Usage authority integration.** Records feed #1596's writer; rollups gain `UsageCoverage`; `pi` marked `aggregate`.
4. **Remove double metering.** Delete the send-level and attempt-usage writes, one-time migration.

## Open questions

1. **`pi` aggregates** — should they land as `granularity: 'aggregate'` `ModelCallAttempt` records so one query answers everything, or stay out of the canonical store with the Usage layer merging two sources? I lean toward the former for a single read path.
2. **Cost resolution point** — resolve `costUsd` at record time (frozen at call time, matching what `RuntimeEvent` already stores) or at projection time from the pricing authority? I lean toward record time: a later price change shouldn't rewrite history.
3. **Migration fidelity** — old telemetry rows have no attempt granularity. Migrate them as `granularity: 'aggregate'` rather than fabricating per-attempt structure?
4. **Retry semantics** — is `retryOf` + shared `traceId` sufficient, or should there be an explicit terminal-attempt marker per logical call for "what did this call finally cost"?

Contributor guide

Open the contributing guide

Research direction

Start with the proposed contract-and-tests step, then read ProviderRequestAttemptRecord and ProviderRequestTracker in provider-request-telemetry.ts, AgentRunEvent registration in core/agent-run.ts, and the referenced runtime-kernel.ts paths. Review the open questions before implementation; done means the canonical record, validation, projection helpers, and all four call paths have an agreed design and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.