erigontech / erigontech/erigon

Testing restart instability

Open
#19,669 1 comment 0 reactions 0 assignees View on GitHub
QA tech debt reduction
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

## Summary

During mainnet sync testing (fresh sync, restart after manual cleanup, branch switching), we encountered several issues that make the sync/restart process fragile and hard to diagnose. These were found on a minimal prune mode mainnet node.

## Issues

### 1. Stale `preverified.toml` prevents re-download (High)

**Symptom**: After manually deleting/modifying snapshot files, the downloader logs `Skipping SyncSnapshots, local preverified` and doesn't re-download missing files.

**Root cause**: `preverified.toml` is a cached file telling the downloader everything is already synced. It persists after file deletion.

**Workaround**: Manually delete `snapshots/preverified.toml` before restart.

**Suggested fix**: The downloader should detect missing files even when `preverified.toml` exists, or `preverified.toml` should be invalidated when referenced files are missing.

### 2. Minimal prune mode downloads full E3 state snapshots on fresh sync (High)

**Symptom**: `--prune.mode=minimal` downloaded 1.3TB (domain, history, idx, accessor) instead of ~291GB (blocks only).

**Root cause**: `buildBlackListForPruning` in `db/snapshotsync/snapshotsync.go` relies on `IterateFrozenBodies` to compute prune boundaries. On a fresh datadir with no frozen bodies, the iteration returns nothing, so `minStepToDownload` stays at `MaxUint32` and nothing gets blacklisted.

**Workaround**: Use `--snap.skip-state-snapshot-download` flag.

**Suggested fix**: The blacklist logic needs a fallback for empty datadirs — if there are no frozen bodies, minimal mode should skip E3 state snapshots entirely.

### 3. "No metadata yet" files stall download startup (Medium)

**Symptom**: Downloader stuck resolving metadata for 4-11 files, sometimes for 10+ minutes, with 0 download rate during this time.

**Root cause**: Some torrent files (especially for newer/smaller segments) take longer to resolve metadata from peers. Downloads for files with resolved metadata appear blocked until all metadata resolves.

**Suggested fix**: The downloader should start downloading files with resolved metadata while waiting for others.

### 4. `IterateFrozenBodies` crash: slice bounds out of range (Medium)

**Symptom**: `panic: runtime error: slice bounds out of range [36:34]` at `block_reader.go:1310` during initial snapshot processing.

**Root cause**: Unknown — transient. Integrity check passed, restart succeeded past the crash point. Could be a race condition during snapshot loading.

**Suggested fix**: Needs investigation — may be a race condition in snapshot file loading during startup.

### 5. `--caplin.blocks-archive` filter checks wrong filename (Low)

**Symptom**: The `NoCaplin` filter in `db/snapshotsync/snapshotsync.go:398` skips files containing `"blobsidecars"` but actual filenames use `"blocksidecars"`. The filter is a no-op for these files.

**Impact**: Benign — means `blocksidecars` are always downloaded regardless of the flag, which is actually what you want for normal sync. But the filter logic is incorrect.

**Suggested fix**: Update the filter string to match actual filenames, or reconsider whether the filter is needed at all.

## Environment

- Machine: AMD EPYC 4244P, 12 cores, 125GB RAM, 7TB NVMe
- Config: `--prune.mode=minimal`, `--chain=mainnet`
- Branch: main (commit 76f310a266)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.