[Proposal] [Experiment] Snapshots Evolution: From Sequential Reexecution to Distributed Reconstruction

### Analogy

How was the first compiler compiled? It wasn't — it was hand-written in assembly. But eventually compilers reached the point where they could bootstrap themselves, and nobody ever wrote a compiler in pure assembly again. That moment unlocked evolution in directions that were impossible before.

Today, Erigon's snapshot production is like writing a compiler in assembly every time: we reexecute from block 0. This has become a clear bottleneck — not just for performance, but for our ability to evolve the data model and our own workflows.

### The Problem

With Erigon 3, we have mature, published snapshots for mainnet and major chains. Yet our tooling still treats them as disposable artifacts that must be regenerated from scratch. This creates several pain points:

1. **Bug investigation doesn't scale.** A data corruption is discovered. We don't know when or where it was introduced. The only option is reexecuting from genesis — days or weeks on mainnet.

2. **New optional data flags require full reexecution.** Adding a derived data product (e.g., a new index, a new history variant) means running the entire chain again, even though the new data depends only on existing state + block data that we already have.

3. **No incremental integrity verification.** We can't sample a range of steps, verify them against a known-good baseline, and bisect to find where corruption began.

4. **Sequential reexecution only.** Today, reexecution must proceed sequentially from genesis — block N depends on the state produced by block N-1. This makes the process inherently single-machine and single-threaded in terms of progress. We can't parallelize across step ranges, and we can't distribute work across multiple nodes. Every operational workflow (bug investigation, data regeneration, new flag production) is bottlenecked by this sequential dependency.

### Existing Tooling: Scattered but Already Halfway There

We already have a significant number of tools that perform pieces of this workflow — they're just scattered across two binaries with ad-hoc interfaces:

Tools like `integration history rebuild`, `integration compact_domains`, `integration commitment rebuild`, and `erigon seg integrity/verify-state/verify-history/diff/unmerge/squeeze/retire/step-rebase` already know how to operate on step ranges, compare files, rebuild domains, and check integrity.

Full inventory of existing tools

**`integration` binary:**
- `history rebuild` — regenerate `.ef`/`.v` files for a domain from a step range
- `history print` / `history distribution` — inspect history entries within step ranges
- `compact_domains` — deduplicate `.kv` domain files for a step range
- `commitment rebuild` — rebuild commitment domain from other domains
- `stage_exec` / `loop_exec` — reexecute blocks (but always as part of the full pipeline)
- `state_stages` — run stages with optional unwind cycles and integrity checks

**`erigon seg` (aka `erigon snapshots`):**
- `integrity` / `verify-state` / `verify-history` — integrity checks with sampling and step ranges
- `diff` — compare two snapshot files
- `compareIdx` — compare two accessor files
- `unmerge` — split a merged file back to 1-step files
- `squeeze` — domain squeeze operation
- `retire` — create snapshot files from a block number
- `step-rebase` — rebase snapshots to a new step size
- `rm-state-snapshots` — remove files by step or domain
- `rollback-snapshots-to-block` — rollback to a given block

What's missing is a **unifying interface** that makes these operations composable and discoverable. Today, each tool has its own flags, its own way of specifying ranges, its own assumptions about what's loaded. A developer who needs to "rebuild steps 500-600 for the accounts domain and verify the result" has to know which combination of commands to use and in what order.

This proposal is about repurposing these existing capabilities around a common interface where:
- Operations are called uniformly (reconstruct, compare, merge, verify)
- Domain/entity types are discoverable and pluggable — adding a new domain means implementing the interface, and all existing tools work with it automatically
- Step ranges are the universal unit of work across all operations

### Proposal

**Treat published Erigon 3 snapshots as "golden images" — bootstrapped, trusted baselines — and rework all production tooling to operate on top of them.**

A **golden image** is the snapshot set published with the latest stable Erigon release — the same data that new nodes download via torrent/webseed to bootstrap. It represents the canonical, validated state of the chain at the time of that release.

Core assumptions:
- Golden images are **trusted until proven otherwise**, with tools to prove or disprove their correctness at any granularity
- Everything has mechanisms for **integrity verification and correction** without syncing from genesis
- Syncing from genesis becomes the **last resort**, reserved for bootstrapping new chains

#### 1. Minimum Unit of Reexecution: 1 Step

Erigon should support reconstructing an arbitrary range of 1 or more steps. Given a golden image and a step range `[a, b)`, the system produces a fresh set of domain files, history files, and indexes for exactly that range.

This is the foundational primitive that enables everything else.

#### 2. Step Bundles as First-Class Objects

Introduce the concept of a **step bundle** — a logical unit of 1+ consecutive steps with well-defined operations modeled as interfaces:

**Equality** — given two step bundles covering the same range, determine whether they are semantically equivalent.

- Equality is defined at the **semantic level**: two bundles are equal if they encode the same logical state, regardless of physical representation. This is the general property that all operations must respect.
- When the encoding and compression settings are identical, semantic equality reduces to **byte equality** — this is the fast path for the common case of validating a reproduction against the golden image.
- The semantic distinction matters for safe migrations: if we add `.ef` optimizations, page-level compression for history, or any encoding change, the raw bytes differ but the data is semantically equivalent. The equality interface validates such conversions without reexecution.
- If a reproduction with the same settings produces different bytes, a bug exists. This property enables **bisection**: binary-search across step ranges to pinpoint exactly where corruption was introduced, then rebuild only from that point forward.

**Mergeability** — the existing merge logic, but modeled as a proper algebraic operation on step bundles. Given bundles `[a, b)` and `[b, c)`, produce `[a, c)`. This is already implemented in practice; the proposal is to formalize it behind an interface so it composes cleanly with the other operations.

### Benefits

**Predictable error prevention**
- Integrity checks can run continuously on published snapshots, catching drift before it propagates.

**Better error correction**
- Bisect to the exact step range where corruption began. Rebuild from there. No genesis reexecution.

**Modular workflow for new data products**
- Adding a new optional flag becomes: "for each step bundle in the golden image, derive the new data." Parallelizable, incremental, restartable.

**Distributed data production**
- The monolithic "one powerful machine reexecutes everything" model is replaced by:
1. Allocate N ephemeral machines
2. Each downloads the current golden image
3. Split the step range into N bundles of work
4. Each machine independently reexecutes its range and produces new files
5. Aggregate and merge

**LLM-friendly tooling**
- Well-defined operations (reconstruct, compare, merge) with clear inputs/outputs are composable by both humans and automated agents, enabling more powerful CI/CD and debugging workflows.

### Implementation Milestones

#### Milestone 1a: Single-Machine Step-Range Reexecution for Optional Data

**Goal:** Prove the core reconstruct-and-validate loop on a single machine using optional/derived data.

**POC target: persisted receipts.**
- Reexecute an arbitrary step range against a golden image and regenerate the receipt domain files for that range.
- Validate that the regenerated data matches the existing golden data (equality property).

Receipts are the ideal proving ground: they are derived entirely from execution and don't affect the state trie, so a failure is recoverable without consequences.

#### Milestone 1b: Distributed Execution and Aggregation

**Goal:** Extend 1a to distributed execution across multiple machines.

- Distribute step ranges as independent jobs across multiple ephemeral Erigon instances on different machines.
- Dispatch, wait for completion, validate each worker's output.
- Aggregate results and replace the corresponding data in the master dataset.

This milestone proves the full loop — reconstruct, validate, distribute, aggregate — and validates that step bundles compose correctly when produced independently.

#### Milestone 2: Reexecute Steps and Regenerate Core State

**Goal:** Apply the validated architecture to the core state domains.

- Regenerate accounts, storage, code, and commitment domain/history/index files using the same reconstruct-validate-aggregate workflow proven in Milestone 1.
- At this point, Erigon should be able to **self-heal**: given a golden image with detected corruption in a step range, the node can autonomously reconstruct the affected steps, validate the new output, and replace the corrupted data — without resyncing from genesis.
- Integrity verification becomes a continuous process: sample step bundles, compare against a fresh reexecution, bisect on mismatch, rebuild from the divergence point.