[RFC] Session Inspector: a per-session runtime event trace, replay, and cost-attribution surface
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 715
Description
## Summary
Spin off the **Observability** track of #544 (durable agent platform) into a concrete, buildable surface: a **Session Inspector** — a per-session, causal, replayable view over the `RuntimeEvent` ledger that Maka already records, with per-step latency and cost attribution, compaction/retry/recovery explanation, failure attribution, and search.
This is contract-first and introduces **no new event store, registry, permission path, or execution runtime** — it is a pure projection over the authoritative ledger.
## Problem
Maka already records a rich per-turn causal ledger (`RuntimeEvent`, `packages/core/src/runtime-event.ts`): every LLM call, tool dispatch (`RuntimeEventToolDispatch`), tool result, `thinking`, error, token usage (`RuntimeEventTokenUsage`), permission decision (`RuntimeEventPermissionDecision`), recovery mode (`replay_safe` / `idempotent` / `reconcile` / `reattach`), and stable `runId` / `sessionId` / `turnId` / `invocationId` identity. #1549 made this ledger a **replayable** agent-graph timeline, and #1596 established a **Usage/Pricing** authority.
None of this causal detail is explorable by the user. The only analytics surface today is the **global, request-level** Usage dashboard in Settings (`apps/desktop/src/renderer/settings/usage-settings-page.tsx`): time-range + model filter, aggregate cost/token/cache metric cards, and requests/providers/models tabs. It answers *"how much did I spend across everything"*, not *"what happened inside this run, and why"*.
So when a run is slow, expensive, or wrong, the user cannot see:
- the **causal step sequence** of a turn (LLM → tool dispatch → result → next LLM), with per-step latency and token/cost delta;
- **where and why context compacted** (`history-compact-ledger.ts` boundaries are invisible today);
- **which tool calls retried, failed, or were replayed on resume** (recovery modes are recorded but never surfaced);
- **failure attribution** — e.g. "this turn failed because tool X errored after 2 retries, then the supervisor context overflowed and compacted";
- any way to **search** the trace or **step through** the replayable timeline.
This is exactly the #544 Track-1 Observability bullet ("live event traces, LLM/tool/cost/compaction/retry/task state, failure attribution, replay, and searchable inspection views"), and it is unbuilt at the read-model and UI layers.
### Positioning
`openai/codex`'s observability is a terminal `/status` (context-window %, token totals, session info) plus TUI cost lines. Maka already pays to record a far richer ledger; the missing piece is making it explainable. This is a place where Maka can lead the reference implementations rather than match them, and it directly serves the "explainable / observable" pillar of the durable agent platform.
## Non-goals (to bound scope)
- **Not** a second event store, registry, or execution path. The Inspector is a **pure projection over the authoritative `RuntimeEvent` ledger** and never writes runtime state.
- **Not** a replacement for Settings → Usage. That stays the account-level aggregate; the Inspector is per-session and causal. Both consume the same Usage/Pricing authority (#1596) — no second pricing source.
- **Not** a new permission or authorization path. Visibility ≠ authority; it reads terminal facts, it does not re-run tools.
- **Not** the CLI `/context` diagnostics view (#1423): that is a read-only context snapshot; this is a desktop causal timeline. Shared vocabulary is fine; different surfaces.
## Architecture: three layers, three PRs
### PR 1 — `RuntimeEventTrace` read model (pure, fully unit-tested)
A pure projection in `packages/runtime` (sibling to `runtime-event-read-model.ts`, which projects events → *stored chat messages*; this projects events → an *analytics trace*). No I/O, no React, deterministic.
Sketch (not final API):
```ts
// packages/runtime/src/runtime-event-trace.ts
export interface TraceStep {
id: string; // invocationId
kind: 'llm' | 'tool' | 'compaction' | 'retry' | 'permission' | 'recovery' | 'error';
turnId: string; runId: string;
startedAt: number; endedAt?: number; durationMs?: number;
tokens?: RuntimeEventTokenUsage;
costUsd?: number; // via the #1596 Usage/Pricing authority
toolName?: string;
status: RuntimeEventStatus;
recoveryMode?: ToolRecoveryMode;
failure?: { code: string; attributedToStepId?: string };
children?: TraceStep[]; // graph/swarm nesting from the #1549 timeline
}
export interface SessionTrace {
sessionId: string;
turns: TurnTrace[]; // grouped by turnId
totals: { durationMs: number; costUsd: number; tokens: RuntimeEventTokenUsage; compactions: number; retries: number };
}
export function projectRuntimeEventsToTrace(events: readonly RuntimeEvent[], pricing: PricingAuthority): SessionTrace;
export function attributeTurnFailure(turn: TurnTrace): FailureAttribution | undefined;
export function searchTrace(trace: SessionTrace, query: TraceQuery): TraceStep[];
```
Reuses existing seams: `classifyRuntimeEventTerminalFact`, `TERMINAL_RUNTIME_EVENT_STATUSES`, `history-compact-ledger` boundaries, the #1549 replayable timeline reconstruction, and the #1596 pricing authority.
Tests (deterministic, node): step grouping by turn; latency/token/cost deltas; compaction-boundary detection; retry & recovery classification; failure attribution across a failed-tool → overflow → compact chain; search predicates; graph/swarm nesting; empty / partial / replayed-ledger edge cases.
PR 1 is behavior-preserving and fully pure — mergeable on its own with no UI.
### PR 2 — Desktop Session Inspector panel + read-only IPC
A renderer panel (alongside `task-ledger-panel.tsx`) that consumes `SessionTrace` for the active session and renders a **vertical causal timeline**: each step with icon, kind, duration bar, token/cost delta, and expandable detail (tool args/result refs, permission decision, recovery reason, compaction summary). Failure steps highlight and link to the attributed cause.
- Read-only IPC that projects the trace; extend the existing `session-event-health.ts` / `app-shell-session-events.ts` event-stream subscription rather than opening a new channel.
- Live-updates as the turn runs (reuse `hasInFlightToolActivity` / stream-snapshot signals).
- Theme-aware; matches the current design-system panel grammar.
### PR 3 — Search + replay step-through
- Trace search over `searchTrace` (jump to compaction, steps > $X, failed tools, by tool name).
- **Replay step-through** over the #1549 replayable ledger: scrub/step the timeline to inspect reconstructed state at each boundary (read-only; no re-execution).
## Open questions for maintainers
1. Panel home: a new right-rail Inspector tab, or an expansion of the session workbar next to the Task ledger?
2. Should `RuntimeEventTrace` live in `packages/runtime` (next to the read model) or `packages/core` (next to `runtime-event.ts`)? Leaning `runtime`, since it depends on the pricing authority.
3. Cost-attribution granularity: per-invocation (LLM request) is exact; per-tool-step cost is derived — acceptable to carry cost on the enclosing LLM step and show tool steps at $0?
4. Does the replayable timeline (#1549) already expose a public reconstruction entry point PR 1 should consume, or should PR 1 define that read boundary?
## Scope discipline
PR 1 is pure and independently mergeable. PR 2/3 are additive desktop surfaces gated behind the panel. Nothing changes the ledger, pricing authority, permission path, or execution runtime.
Contributor guide
Assessment
This issue has not been assessed yet.