EventV2 cold history duplicates large snapshot payloads and amplifies SQLite storage/reads

Description

This is a mitigation/design follow-up to #33356, #46833, #47223, #47729, and #43551. Those issues already document unbounded event growth from large durable snapshots. I am not trying to add another independent reproduction.

I tested a cold-history storage layer in my downstream fork. The result suggests that EventV2 can keep its logical append-only model while storing old payloads more efficiently.

Current upstream behavior, verified against dev, is relevant here:

• durable events store the payload in event.data as JSON text;
• message.updated.1 can carry the full message info, including summary.diffs;
• repeated updates can therefore store the same or near-identical large field many times;
• historical aggregate reads materialize complete event rows, including event.data.

The storage cost can therefore become a read cost too.

Proposed storage model

Keep new event payloads inline. After history leaves a protected hot tail, move large payloads to an aggregate-local content-addressed value table and replace event.data with a small reference.

flowchart LR
    A[New EventV2 event] --> B[Inline JSON hot tail]
    B --> C{Eligible for cold sealing?}
    C -- No --> B
    C -- Yes --> D[Background sealer]
    D --> E{Same payload in aggregate?}
    E -- Yes --> F[Reuse canonical value]
    E -- No --> G[Compress canonical value]
    G --> H[(event_value)]
    F --> H
    H --> I[Small cdbRef in event.data]

On read, references are resolved in a batch and the original logical payload is restored before EventV2 schema decoding.

sequenceDiagram
    participant R as Event read
    participant E as event
    participant V as event_value
    participant C as decoded cache
    R->>E: Read ordered rows
    E-->>R: Inline rows + refs
    R->>C: Check referenced values
    R->>V: Fetch cache misses in batch
    V-->>R: Canonical compressed bytes
    R->>R: Decode, validate, rehydrate

Important properties of the downstream prototype:

• event ID, type, aggregate ID, and sequence do not change;
• the newest 256 events of an active session stay inline;
• the remaining tail becomes eligible after a 1-hour cooling period;
• exact deduplication is scoped to (aggregate_id, sha256);
• each canonical value stores raw_len and SHA-256;
• referenced event payloads fail closed on missing/corrupt values;
• compression/decompression can use worker pools;
• background writes use small transactions and yield on SQLITE_BUSY;
• large SQLite freelists are reclaimed with incremental vacuum rather than a live full VACUUM;
• jumbo/repetitive delta inputs have explicit bounds and bypasses.

Reference implementation

Pinned downstream snapshot:
https://github.com/thelabcorner/openfork/commit/7cd1b421b33e55512d694c9b830d11f2aac2aa27

Key files:

• Sealing, hot-tail policy, exact dedup, reclaim: https://github.com/thelabcorner/openfork/blob/7cd1b421b33e55512d694c9b830d11f2aac2aa27/packages/core/src/database/chunk-sealer.ts
• Frame format and codecs: https://github.com/thelabcorner/openfork/blob/7cd1b421b33e55512d694c9b830d11f2aac2aa27/packages/core/src/database/json-codec.ts
• EventV2 reference resolution/cache: https://github.com/thelabcorner/openfork/blob/7cd1b421b33e55512d694c9b830d11f2aac2aa27/packages/core/src/event.ts
• Schema/setup: https://github.com/thelabcorner/openfork/blob/7cd1b421b33e55512d694c9b830d11f2aac2aa27/packages/core/src/database/chunkdb.ts
• Worker packaging: https://github.com/thelabcorner/openfork/blob/7cd1b421b33e55512d694c9b830d11f2aac2aa27/packages/opencode/script/build-node.ts
• Crash/restart/reclaim tests: https://github.com/thelabcorner/openfork/blob/7cd1b421b33e55512d694c9b830d11f2aac2aa27/packages/core/test/database/chunkdb-crash.test.ts
• Delta regression tests: https://github.com/thelabcorner/openfork/blob/7cd1b421b33e55512d694c9b830d11f2aac2aa27/packages/core/test/database/json-codec-delta.test.ts

I do not suggest copying the downstream implementation line for line. My fork has diverged. The useful parts are the storage model, invariants, failure cases, and measurements.

Same-corpus storage validation

I created an inline-JSON comparison database from the sealed database and verified the restored payloads directly.

• 71,333 transformed events checked
• 8,147,046,149 restored payload bytes
• 0 payload mismatches
• 61,151 canonical values passed SHA-256 validation
• 684,690 event IDs on both sides
• 0 aggregate/sequence/type mismatches
• 136/136 non-ChunkDB schema/index objects matched after excluding the intentional ChunkDB objects

Settled file sizes:

• Inline JSON: 10,806,681,600 bytes (10.0645 GiB)
• ChunkDB: 3,696,328,704 bytes (3.4425 GiB)
• Reduction: 65.7959%

The seal journal represented 8,147,046,149 raw historical payload bytes with 621,486,236 stored bytes in the ChunkDB representation.

Read benchmark

The harness measured full-aggregate EventTable materialization plus the production rehydrateEvents() function. It did not include the final EventV2 schema-decode wrapper. Windows file cache was not flushed, so these are not cold-disk measurements.

For a saved 12-session cohort of approximately 5.84 MB to 114.86 MB logical payload:

• first decoded load: ChunkDB won 1/12; geometric relative performance was 0.5986x, about 1.67x slower;
• subsequent loads: ChunkDB won 12/12; median speedup was 5.8213x and geometric-mean speedup was 4.5244x.

The first-read cost is still an optimization target. Reference lookup, hashing, worker dispatch, and decompression are not free.

I think the upstream design question is whether cold EventV2 payload storage should become a separate physical layer from the logical event model. The existing growth reports show the problem. This prototype is evidence that compaction/dedup can address it without deleting historical events.

_{Disclosure: my coding agent helped draft this issue from my local implementation, tests, and notes. I reviewed the technical claims, but there can still be mistakes or missing upstream context. If something looks wrong, please ask and I will verify it.}

Plugins

Not relevant. This is in the core EventV2 storage path.

OpenCode version

Upstream dev was cross-checked during this report. The downstream reference implementation is pinned at 7cd1b421b33e55512d694c9b830d11f2aac2aa27.

Steps to reproduce

A representative workload is a long session where message.updated.1 repeatedly carries a large summary.diffs value.

1. Run a session that generates large repository diffs and several durable message updates.
2. Inspect event rows for that session and group or sum length(data) by event type.
3. Compare repeated message.updated.1 payloads and their summary.diffs fields.
4. Reopen/read the historical session and observe that full event rows must be materialized for replay.

The linked upstream issues contain real databases and larger examples of the same behavior.

Screenshot and/or share link

Not applicable.

Operating System

Benchmark environment: Windows 11.

Terminal

Not relevant.

Start with the linked upstream growth issues and compare the proposed model with the pinned reference files: chunk-sealer.ts, json-codec.ts, event.ts, and chunkdb.ts. Review chunkdb-crash.test.ts and json-codec-delta.test.ts for stated failure cases and invariants. Done means an agreed upstream design, scope, and validation plan for cold payload storage rather than a direct copy of the downstream implementation.