apache / apache/maka

[RFC] Session Inspector: a per-session runtime event trace, replay, and cost-attribution surface

Open
#1,625 10 comments 0 reactions 1 assignee Claimed by @ARE404 View on GitHub
enhancement
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
715

Description

## Summary

Spin off the **Observability** track of #544 (durable agent platform) into a concrete, buildable surface: a **Session Inspector** — a per-session, causal, replayable view over the `RuntimeEvent` ledger that Maka already records, with per-step latency and cost attribution, compaction/retry/recovery explanation, failure attribution, and search.

This is contract-first and introduces **no new event store, registry, permission path, or execution runtime** — it is a pure projection over the authoritative ledger.

## Problem

Maka already records a rich per-turn causal ledger (`RuntimeEvent`, `packages/core/src/runtime-event.ts`): every LLM call, tool dispatch (`RuntimeEventToolDispatch`), tool result, `thinking`, error, token usage (`RuntimeEventTokenUsage`), permission decision (`RuntimeEventPermissionDecision`), recovery mode (`replay_safe` / `idempotent` / `reconcile` / `reattach`), and stable `runId` / `sessionId` / `turnId` / `invocationId` identity. #1549 made this ledger a **replayable** agent-graph timeline, and #1596 established a **Usage/Pricing** authority.

None of this causal detail is explorable by the user. The only analytics surface today is the **global, request-level** Usage dashboard in Settings (`apps/desktop/src/renderer/settings/usage-settings-page.tsx`): time-range + model filter, aggregate cost/token/cache metric cards, and requests/providers/models tabs. It answers *"how much did I spend across everything"*, not *"what happened inside this run, and why"*.

So when a run is slow, expensive, or wrong, the user cannot see:

- the **causal step sequence** of a turn (LLM → tool dispatch → result → next LLM), with per-step latency and token/cost delta;
- **where and why context compacted** (`history-compact-ledger.ts` boundaries are invisible today);
- **which tool calls retried, failed, or were replayed on resume** (recovery modes are recorded but never surfaced);
- **failure attribution** — e.g. "this turn failed because tool X errored after 2 retries, then the supervisor context overflowed and compacted";
- any way to **search** the trace or **step through** the replayable timeline.

This is exactly the #544 Track-1 Observability bullet ("live event traces, LLM/tool/cost/compaction/retry/task state, failure attribution, replay, and searchable inspection views"), and it is unbuilt at the read-model and UI layers.

### Positioning

`openai/codex`'s observability is a terminal `/status` (context-window %, token totals, session info) plus TUI cost lines. Maka already pays to record a far richer ledger; the missing piece is making it explainable. This is a place where Maka can lead the reference implementations rather than match them, and it directly serves the "explainable / observable" pillar of the durable agent platform.

## Non-goals (to bound scope)

- **Not** a second event store, registry, or execution path. The Inspector is a **pure projection over the authoritative `RuntimeEvent` ledger** and never writes runtime state.
- **Not** a replacement for Settings → Usage. That stays the account-level aggregate; the Inspector is per-session and causal. Both consume the same Usage/Pricing authority (#1596) — no second pricing source.
- **Not** a new permission or authorization path. Visibility ≠ authority; it reads terminal facts, it does not re-run tools.
- **Not** the CLI `/context` diagnostics view (#1423): that is a read-only context snapshot; this is a desktop causal timeline. Shared vocabulary is fine; different surfaces.

## Architecture: three layers, three PRs

### PR 1 — `RuntimeEventTrace` read model (pure, fully unit-tested)

A pure projection in `packages/runtime` (sibling to `runtime-event-read-model.ts`, which projects events → *stored chat messages*; this projects events → an *analytics trace*). No I/O, no React, deterministic.

Sketch (not final API):

```ts
// packages/runtime/src/runtime-event-trace.ts
export interface TraceStep {
id: string; // invocationId
kind: 'llm' | 'tool' | 'compaction' | 'retry' | 'permission' | 'recovery' | 'error';
turnId: string; runId: string;
startedAt: number; endedAt?: number; durationMs?: number;
tokens?: RuntimeEventTokenUsage;
costUsd?: number; // via the #1596 Usage/Pricing authority
toolName?: string;
status: RuntimeEventStatus;
recoveryMode?: ToolRecoveryMode;
failure?: { code: string; attributedToStepId?: string };
children?: TraceStep[]; // graph/swarm nesting from the #1549 timeline
}

export interface SessionTrace {
sessionId: string;
turns: TurnTrace[]; // grouped by turnId
totals: { durationMs: number; costUsd: number; tokens: RuntimeEventTokenUsage; compactions: number; retries: number };
}

export function projectRuntimeEventsToTrace(events: readonly RuntimeEvent[], pricing: PricingAuthority): SessionTrace;
export function attributeTurnFailure(turn: TurnTrace): FailureAttribution | undefined;
export function searchTrace(trace: SessionTrace, query: TraceQuery): TraceStep[];
```

Reuses existing seams: `classifyRuntimeEventTerminalFact`, `TERMINAL_RUNTIME_EVENT_STATUSES`, `history-compact-ledger` boundaries, the #1549 replayable timeline reconstruction, and the #1596 pricing authority.

Tests (deterministic, node): step grouping by turn; latency/token/cost deltas; compaction-boundary detection; retry & recovery classification; failure attribution across a failed-tool → overflow → compact chain; search predicates; graph/swarm nesting; empty / partial / replayed-ledger edge cases.

PR 1 is behavior-preserving and fully pure — mergeable on its own with no UI.

### PR 2 — Desktop Session Inspector panel + read-only IPC

A renderer panel (alongside `task-ledger-panel.tsx`) that consumes `SessionTrace` for the active session and renders a **vertical causal timeline**: each step with icon, kind, duration bar, token/cost delta, and expandable detail (tool args/result refs, permission decision, recovery reason, compaction summary). Failure steps highlight and link to the attributed cause.

- Read-only IPC that projects the trace; extend the existing `session-event-health.ts` / `app-shell-session-events.ts` event-stream subscription rather than opening a new channel.
- Live-updates as the turn runs (reuse `hasInFlightToolActivity` / stream-snapshot signals).
- Theme-aware; matches the current design-system panel grammar.

### PR 3 — Search + replay step-through

- Trace search over `searchTrace` (jump to compaction, steps > $X, failed tools, by tool name).
- **Replay step-through** over the #1549 replayable ledger: scrub/step the timeline to inspect reconstructed state at each boundary (read-only; no re-execution).

## Open questions for maintainers

1. Panel home: a new right-rail Inspector tab, or an expansion of the session workbar next to the Task ledger?
2. Should `RuntimeEventTrace` live in `packages/runtime` (next to the read model) or `packages/core` (next to `runtime-event.ts`)? Leaning `runtime`, since it depends on the pricing authority.
3. Cost-attribution granularity: per-invocation (LLM request) is exact; per-tool-step cost is derived — acceptable to carry cost on the enclosing LLM step and show tool steps at $0?
4. Does the replayable timeline (#1549) already expose a public reconstruction entry point PR 1 should consume, or should PR 1 define that read boundary?

## Scope discipline

PR 1 is pure and independently mergeable. PR 2/3 are additive desktop surfaces gated behind the panel. Nothing changes the ledger, pricing authority, permission path, or execution runtime.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.