erigontech / erigontech/erigon
cl/snapshots: verify and repair blob snapshots before publication
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
# Rationale
Blob-sidecar snapshots can become incomplete or internally inconsistent before publication. Unlike derived indexes, blob contents cannot be reconstructed from commitments alone, and the P2P availability window is finite. Publishing or seeding an incomplete range can therefore turn a recoverable local gap into permanently unavailable historical data.
The current paths catch some count mismatches while dumping, but there is no end-to-end snapshotter workflow that validates an at-rest blob snapshot set, repairs recoverable gaps from peers, re-validates it, and refuses publication when repair fails. Retrying the same range indefinitely is not an acceptable integrity policy.
This issue covers the snapshot producer/publisher side. The operator workflow that restores from official snapshots is tracked separately.
# Implementation
Add a snapshotter integrity-and-repair workflow for blob-sidecar snapshots:
- Determine the expected blob data for every canonical beacon block in the target range, including slot, block root, sidecar count/indexes, and fork-specific requirements.
- Validate snapshot range coverage, readable segment records, index-to-segment consistency, commitment counts, sidecar identity, and KZG proofs at rest.
- Report the exact affected ranges and distinguish missing data, corrupt data, missing indexes, and unavailable source data.
- For recoverable gaps, reuse the historical blob backfill path to fetch data from peers. Deneb blob sidecars and Fulu data-column reconstruction must use canonical block roots.
- Rebuild and atomically replace the affected immutable snapshot range and its derived index instead of mutating a published segment in place.
- Re-run all checks after repair.
- If any required sidecar remains missing or invalid, return a hard failure and prevent the affected files from being published or seeded.
- Bound retries and resource consumption; an unavailable peer source must not cause an endless retry loop.
The command should be available from the main `erigon` binary, preferably as part of the existing `erigon snapshots` command family, and reusable by the automated snapshot publication pipeline.
## Acceptance criteria
- A complete fixture passes without rewriting files.
- Missing sidecar, wrong sidecar count, invalid proof/commitment, wrong canonical root, unreadable segment, and missing/corrupt index fixtures are detected.
- Recoverable missing data is fetched, the affected range is rebuilt, and a second verification passes.
- When peers cannot provide required data, the command exits non-zero and the affected range is not published or seeded.
- Repair is restart-safe and does not leave a partially visible replacement range.
- Tests cover zero-blob blocks, fork boundaries, empty slots, range boundaries, and both Deneb and Fulu recovery paths.
## Related
- #23024 tracks Caplin snapshot lifecycle parity and explicitly lists blob-sidecar count/KZG integrity checks as unfiled work.
- #23224 demonstrates that a single missing sidecar can wedge blob antiquation.
- #23896 tracks canonical-root bugs in the current historical blob backfill path.
- #23752 tracks retention/backfill-window behavior that may make peer repair unavailable.
- #23909 is an operator report of missing historical blobs and very slow recovery.
- #23912 tracks the operator workflow that repairs a datadir from official snapshots.
Contributor guide
Assessment
This issue has not been assessed yet.