erigontech / erigontech/erigon
downloader: extract preverified.toml filters into a reusable, testable function
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
## Problem
The filters applied to `preverified.toml` to determine which snapshots to download (based on chain, archive/prune mode, step size, and other parameters) are currently:
- **Reproduced in multiple places** across the codebase rather than sharing a single implementation.
- **Only exercisable through the sync stage**, which causes side effects and makes isolated testing impractical.
- **Untested across the full input matrix** — renames, incorrect filter logic, and entire modes being silently dropped have occurred and gone undetected until a user hits the problem.
## Proposed change
Extract the filtering logic into a standalone, pure function (or small package) that:
1. Takes `preverified.toml` (or its parsed representation) plus a set of parameters (chain, prune/archive mode, step size, any other flags that affect selection) and returns the filtered file list.
2. Has no sync-stage side effects — it can be called from tests, CLI subcommands, or future tooling without starting a sync.
3. Replaces all existing call sites that currently duplicate this logic.
## Testing
With the filter as a standalone function, add a table-driven test that runs it across a matrix of inputs:
- Chains (mainnet, gnosis, sepolia, …)
- Archive vs. minimal vs. other prune modes
- Step sizes
- Any other parameters that alter filter behaviour
The test should assert that the output file list is non-empty, contains expected file types, and does not unexpectedly drop known segments. If total byte size can be derived from the preverified metadata, include it as a regression signal — though this may not be possible depending on what the toml records.
## Benefits
- **Detect renames early**: if a segment type is renamed in the preverified set, the filter test catches it before it reaches users.
- **Detect dropped modes**: historically, certain prune/archive combinations have been silently dropped from the filtered output; a matrix test will catch this.
- **Reusability**: the filter can be called from the `erigon snapshots` subcommands (e.g. for #19715 filter recording, or #19714 source selection) and from any future tooling without requiring a live sync context.
- **Single source of truth**: eliminates the divergence that occurs when filter logic is duplicated across call sites.
## Related issues
- #19715 — record and validate download filters (depends on having a canonical filter function to record from)
- #19714 — preverified source selection on main node
- #19716 — error on extraneous snapshots (filter correctness is a prerequisite)
- #19717 — persist preverified.toml at sync start
Contributor guide
Assessment
This issue has not been assessed yet.