erigontech / erigontech/erigon
Testing restart instability
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
## Summary
During mainnet sync testing (fresh sync, restart after manual cleanup, branch switching), we encountered several issues that make the sync/restart process fragile and hard to diagnose. These were found on a minimal prune mode mainnet node.
## Issues
### 1. Stale `preverified.toml` prevents re-download (High)
**Symptom**: After manually deleting/modifying snapshot files, the downloader logs `Skipping SyncSnapshots, local preverified` and doesn't re-download missing files.
**Root cause**: `preverified.toml` is a cached file telling the downloader everything is already synced. It persists after file deletion.
**Workaround**: Manually delete `snapshots/preverified.toml` before restart.
**Suggested fix**: The downloader should detect missing files even when `preverified.toml` exists, or `preverified.toml` should be invalidated when referenced files are missing.
### 2. Minimal prune mode downloads full E3 state snapshots on fresh sync (High)
**Symptom**: `--prune.mode=minimal` downloaded 1.3TB (domain, history, idx, accessor) instead of ~291GB (blocks only).
**Root cause**: `buildBlackListForPruning` in `db/snapshotsync/snapshotsync.go` relies on `IterateFrozenBodies` to compute prune boundaries. On a fresh datadir with no frozen bodies, the iteration returns nothing, so `minStepToDownload` stays at `MaxUint32` and nothing gets blacklisted.
**Workaround**: Use `--snap.skip-state-snapshot-download` flag.
**Suggested fix**: The blacklist logic needs a fallback for empty datadirs — if there are no frozen bodies, minimal mode should skip E3 state snapshots entirely.
### 3. "No metadata yet" files stall download startup (Medium)
**Symptom**: Downloader stuck resolving metadata for 4-11 files, sometimes for 10+ minutes, with 0 download rate during this time.
**Root cause**: Some torrent files (especially for newer/smaller segments) take longer to resolve metadata from peers. Downloads for files with resolved metadata appear blocked until all metadata resolves.
**Suggested fix**: The downloader should start downloading files with resolved metadata while waiting for others.
### 4. `IterateFrozenBodies` crash: slice bounds out of range (Medium)
**Symptom**: `panic: runtime error: slice bounds out of range [36:34]` at `block_reader.go:1310` during initial snapshot processing.
**Root cause**: Unknown — transient. Integrity check passed, restart succeeded past the crash point. Could be a race condition during snapshot loading.
**Suggested fix**: Needs investigation — may be a race condition in snapshot file loading during startup.
### 5. `--caplin.blocks-archive` filter checks wrong filename (Low)
**Symptom**: The `NoCaplin` filter in `db/snapshotsync/snapshotsync.go:398` skips files containing `"blobsidecars"` but actual filenames use `"blocksidecars"`. The filter is a no-op for these files.
**Impact**: Benign — means `blocksidecars` are always downloaded regardless of the flag, which is actually what you want for normal sync. But the filter logic is incorrect.
**Suggested fix**: Update the filter string to match actual filenames, or reconsider whether the filter is needed at all.
## Environment
- Machine: AMD EPYC 4244P, 12 cores, 125GB RAM, 7TB NVMe
- Config: `--prune.mode=minimal`, `--chain=mainnet`
- Branch: main (commit 76f310a266)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Assessment
This issue has not been assessed yet.