apache / apache/maka

fix(runtime): RuntimeEvent ledger is recovery-grade (WAL), not source-of-truth — write-path truncation blocks counterfactual replay

Open
#615 3 comments 0 reactions 1 assignee Claimed by @likun666661 View on GitHub
bug
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
716

Description

## Goal vs reality

The RuntimeEvent ledger is designed to be the **source of truth** for a run: any view (prompt, projection, evidence export) should be rebuildable from it — including *counterfactual* views under different context policies, which is the foundation for replay-based policy evaluation (free pre-filtering of context-budget candidates for the RSI loop) and for AHE evidence fidelity.

An audit (2026-07-07) shows the current implementation is **recovery-grade (WAL)**: it guarantees resuming the view the model actually saw, but cannot rebuild a different view (write path loses pre-policy originals) nor byte-identically rebuild the *same* view (read path depends on live state that is never recorded).

Two criteria define source-of-truth grade:

1. **Facts are written before policy** — no truncation/cleaning/budgeting on the write path; all "show the model less" transforms are read-time projections.
2. **Read is a pure function of (ledger, policy params)** — no live clock/fs/config/registry inputs, or their values are snapshotted per run.

The read side already follows criterion 1's spirit everywhere (all prune/compact mechanisms are projection-time, SHA-verified archives, refuse-to-substitute on archive failure). The violations are at the source and at the edges.

## Write-path violations (criterion 1)

- **F1 (worst): Bash output truncated to 2000 lines / 50 KB before persistence.** `shapeTerminalResult` calls `truncateToolOutput` inside the tool impl (`packages/runtime/src/shell-tools.ts:148-157`, also `builtin-tools.ts:206`; limits in `tool-output.ts:48-49`), so the ledger's canonical record is the truncated view. The header comment (`tool-output.ts:12-14`) documents this as a deliberate choice for the isolated-executor benchmark path. Local path is partially recoverable via `ShellRunStore` 1 MB tail (`shell-run-manager.ts:353-389`, linked by `sourceToolCallId`); the harbor isolated-executor path has no full-output record at all.
- **F2: capture-time 1 MB cap** — `BASH_MAX_RETAINED_CHARS` (`shell-exec.ts:27`); `BashTailBuffer.trim()` (`bash-tail-buffer.ts:58-83`) discards beyond-cap bytes as they stream. The head of a >1 MB output never exists on disk.
- **F3: Pi backend caps tool-result text at 8192 chars** before the `ToolResultEvent` is created (`pi-agent-backend.ts:543-553`).
- **F4 (minor): synthetic tool error text sliced to 4000 chars** before persist (`tool-runtime.ts:87, 735-740`).
- **F5 (accepted exception, keep): secret redaction is write-time** (`shell-tools.ts:146-147` etc.). Correct as-is — secrets must never touch disk. Document as the one sanctioned deviation from criterion 1.

Note the downstream irony: the archive machinery (SHA-256, verified read-back) faithfully preserves content that was already lossy when it reached the ledger.

## Read-path impurities (criterion 2)

The projection core is verified pure (`applyRuntimeEventContextBudget` / `applyRuntimeEventHistoryCompact`, `context-budget.ts:457,566`; tool re-seeding `tool-availability.ts:198-223`). Purity breaks at the edges:

- **Turn tail unrecorded**: live date/platform (`system-prompt/session-environment-prompt.ts:19-20`), git branch (`system-prompt/project-context.ts:22`), live background-shell summary (`ai-sdk-backend.ts:789-792, 2347-2349`) are appended to the user message at send time; the ledger stores only `userInput.text`.
- **System prompt live-read each turn**: instruction files + settings (`system-prompt/workspace-instructions.ts:139`, `cli-system-prompt.ts:20-44`); only `systemPromptHash` is persisted (`request-shape.ts:94`) — drift is detectable but not reproducible.
- **Policy knobs mostly unrecorded**: ~50 `MAKA_CONTEXT_*` env vars gate sub-policies (`context-budget-policy.ts:15-16` ff.); the persisted `ContextBudgetDiagnostic` records names/flags but not ratios, `minRecentTurns`, per-block caps.
- **Tool schema bodies unrecorded**: only `toolSchemaHash` (`request-shape.ts:96-103`).

## Impact

- Counterfactual replay is one-directional today: simulating **tighter** policies over recorded ledgers is faithful; **looser** (e.g. \"what if the Bash cap were 100 KB\") is impossible — the data never existed.
- Byte-exact replay of even the same view is impossible; only hash-level drift detection works.
- Historical runs can never be upgraded retroactively; every day without the capture fix is golden-fixture data not being accumulated.

## Fix plan (priority order)

1. **Move F1/F3 truncation to read-time projection**: persist the full (redacted) tool output as an artifact (reuse the existing SHA-256 archive machinery — `harbor-cell.ts:916-959` proves it exists on the isolated path too), keep a ref in the ledger, make the 50 KB/8 KB budgets projection parameters.
2. **Emit a `request_captured` event at send time** (`ai-sdk-backend.ts:857` area): canonical JSON of resolved system prompt, turn tail, effective `ContextBudgetPolicy` snapshot, tool schema bodies (content-addressed). One change closes all four read-path holes and provides ground truth for the replay test.
3. **Keep the 1 MB capture cap (F2) but record dropped-byte count** so replay knows the uncertainty bound.
4. Fix F4 alongside 1.

## Replay acceptance test (can start now)

1. **Hash-level (no runtime change)**: replay a `maka-eval` fixture ledger through `AiSdkBackend` + `MockLanguageModelV3` (existing harness pattern, `__tests__/ai-sdk-backend.test.ts` asserts on `doStreamCalls[n].prompt`), recompute `computeRequestShapeDiagnostic`, assert `systemPromptHash`/`toolSchemaHash`/`requestShapeHash` equal the recorded `model_stream_started` values.
2. **Content-level** once item 2 lands: byte-compare re-assembled `{system, messages, tools}` against the captured event. New runs only.
3. **Wire-level**: inject a recording `fetch` via `model-factory.ts:19,56,64`, diff literal HTTP bodies.

## Acceptance criteria

- [ ] A recorded run replays offline with byte-identical `{system, messages, tools}` (post item 2, new runs)
- [ ] Full tool outputs recoverable from artifacts regardless of model-facing budget (post item 1)
- [ ] Contract guarded by CI: any future write-path truncation or unrecorded live input turns the replay test red

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.