fix(runtime): RuntimeEvent ledger is recovery-grade (WAL), not source-of-truth — write-path truncation blocks counterfactual replay
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 716
Description
## Goal vs reality
The RuntimeEvent ledger is designed to be the **source of truth** for a run: any view (prompt, projection, evidence export) should be rebuildable from it — including *counterfactual* views under different context policies, which is the foundation for replay-based policy evaluation (free pre-filtering of context-budget candidates for the RSI loop) and for AHE evidence fidelity.
An audit (2026-07-07) shows the current implementation is **recovery-grade (WAL)**: it guarantees resuming the view the model actually saw, but cannot rebuild a different view (write path loses pre-policy originals) nor byte-identically rebuild the *same* view (read path depends on live state that is never recorded).
Two criteria define source-of-truth grade:
1. **Facts are written before policy** — no truncation/cleaning/budgeting on the write path; all "show the model less" transforms are read-time projections.
2. **Read is a pure function of (ledger, policy params)** — no live clock/fs/config/registry inputs, or their values are snapshotted per run.
The read side already follows criterion 1's spirit everywhere (all prune/compact mechanisms are projection-time, SHA-verified archives, refuse-to-substitute on archive failure). The violations are at the source and at the edges.
## Write-path violations (criterion 1)
- **F1 (worst): Bash output truncated to 2000 lines / 50 KB before persistence.** `shapeTerminalResult` calls `truncateToolOutput` inside the tool impl (`packages/runtime/src/shell-tools.ts:148-157`, also `builtin-tools.ts:206`; limits in `tool-output.ts:48-49`), so the ledger's canonical record is the truncated view. The header comment (`tool-output.ts:12-14`) documents this as a deliberate choice for the isolated-executor benchmark path. Local path is partially recoverable via `ShellRunStore` 1 MB tail (`shell-run-manager.ts:353-389`, linked by `sourceToolCallId`); the harbor isolated-executor path has no full-output record at all.
- **F2: capture-time 1 MB cap** — `BASH_MAX_RETAINED_CHARS` (`shell-exec.ts:27`); `BashTailBuffer.trim()` (`bash-tail-buffer.ts:58-83`) discards beyond-cap bytes as they stream. The head of a >1 MB output never exists on disk.
- **F3: Pi backend caps tool-result text at 8192 chars** before the `ToolResultEvent` is created (`pi-agent-backend.ts:543-553`).
- **F4 (minor): synthetic tool error text sliced to 4000 chars** before persist (`tool-runtime.ts:87, 735-740`).
- **F5 (accepted exception, keep): secret redaction is write-time** (`shell-tools.ts:146-147` etc.). Correct as-is — secrets must never touch disk. Document as the one sanctioned deviation from criterion 1.
Note the downstream irony: the archive machinery (SHA-256, verified read-back) faithfully preserves content that was already lossy when it reached the ledger.
## Read-path impurities (criterion 2)
The projection core is verified pure (`applyRuntimeEventContextBudget` / `applyRuntimeEventHistoryCompact`, `context-budget.ts:457,566`; tool re-seeding `tool-availability.ts:198-223`). Purity breaks at the edges:
- **Turn tail unrecorded**: live date/platform (`system-prompt/session-environment-prompt.ts:19-20`), git branch (`system-prompt/project-context.ts:22`), live background-shell summary (`ai-sdk-backend.ts:789-792, 2347-2349`) are appended to the user message at send time; the ledger stores only `userInput.text`.
- **System prompt live-read each turn**: instruction files + settings (`system-prompt/workspace-instructions.ts:139`, `cli-system-prompt.ts:20-44`); only `systemPromptHash` is persisted (`request-shape.ts:94`) — drift is detectable but not reproducible.
- **Policy knobs mostly unrecorded**: ~50 `MAKA_CONTEXT_*` env vars gate sub-policies (`context-budget-policy.ts:15-16` ff.); the persisted `ContextBudgetDiagnostic` records names/flags but not ratios, `minRecentTurns`, per-block caps.
- **Tool schema bodies unrecorded**: only `toolSchemaHash` (`request-shape.ts:96-103`).
## Impact
- Counterfactual replay is one-directional today: simulating **tighter** policies over recorded ledgers is faithful; **looser** (e.g. \"what if the Bash cap were 100 KB\") is impossible — the data never existed.
- Byte-exact replay of even the same view is impossible; only hash-level drift detection works.
- Historical runs can never be upgraded retroactively; every day without the capture fix is golden-fixture data not being accumulated.
## Fix plan (priority order)
1. **Move F1/F3 truncation to read-time projection**: persist the full (redacted) tool output as an artifact (reuse the existing SHA-256 archive machinery — `harbor-cell.ts:916-959` proves it exists on the isolated path too), keep a ref in the ledger, make the 50 KB/8 KB budgets projection parameters.
2. **Emit a `request_captured` event at send time** (`ai-sdk-backend.ts:857` area): canonical JSON of resolved system prompt, turn tail, effective `ContextBudgetPolicy` snapshot, tool schema bodies (content-addressed). One change closes all four read-path holes and provides ground truth for the replay test.
3. **Keep the 1 MB capture cap (F2) but record dropped-byte count** so replay knows the uncertainty bound.
4. Fix F4 alongside 1.
## Replay acceptance test (can start now)
1. **Hash-level (no runtime change)**: replay a `maka-eval` fixture ledger through `AiSdkBackend` + `MockLanguageModelV3` (existing harness pattern, `__tests__/ai-sdk-backend.test.ts` asserts on `doStreamCalls[n].prompt`), recompute `computeRequestShapeDiagnostic`, assert `systemPromptHash`/`toolSchemaHash`/`requestShapeHash` equal the recorded `model_stream_started` values.
2. **Content-level** once item 2 lands: byte-compare re-assembled `{system, messages, tools}` against the captured event. New runs only.
3. **Wire-level**: inject a recording `fetch` via `model-factory.ts:19,56,64`, diff literal HTTP bodies.
## Acceptance criteria
- [ ] A recorded run replays offline with byte-identical `{system, messages, tools}` (post item 2, new runs)
- [ ] Full tool outputs recoverable from artifacts regardless of model-facing budget (post item 1)
- [ ] Contract guarded by CI: any future write-path truncation or unrecorded live input turns the replay test red
Contributor guide
Assessment
This issue has not been assessed yet.