erigontech / erigontech/erigon
Downloader: missing torrent metadata for old files can block node startup, or cause confusing logging
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
## Downloader: missing torrent metadata can block node startup indefinitely, or cuase confusing logging
The issue below was observred while testing and happened several times, while benign from a functional perspective (the node was configured with `snapCfg.Local=true`) in this scenario it is confusing becuase its not clear wether to abandon the run or simply wait.
This gets exacerbated with agentic testing as the agent incorectly assumes the process is not working and attempts to fix the problem. Which wastes agent time. I think the quickets fix is to not produce these logs if we're operating in `snapCfg.Local=true`. But I also think the commentry around v1.0 files is worth fixing with something more than a work around.
`snapCfg.Local=true`
When the downloader encounters snapshot files on disk that lack torrent metadata (`.torrent` files or cached metainfo), the `batch.wait()` function blocks until every torrent in the batch completes. A torrent without metadata can never determine what pieces it needs and therefore can never complete. The only escape is context cancellation, but the sync context has no timeout — it only cancels on shutdown. This means the node can hang indefinitely on startup.
This is a long-standing issue that has been reported multiple times over the years but has never been fixed at the code level:
- #4034 — "Erigon stuck in 'Waiting for torrents metadata' phase" (May 2022, workaround: `rm *.torrent` + restart)
- #4154 — "Stuck at 'Waiting for torrents metadata: 0/87'" (2022)
- #5042 — "Erigon must self-heal from 'Waiting for torrents metadata: 121/122' state" (Aug 2022, closed as stale)
- #7302 — "[1/15 Snapshots] Waiting for torrents metadata" (2023)
- #8861 — "[Sepolia] Stuck syncing with 'Waiting for torrents metadata: 38/39'" (2023)
All prior reports were closed either as stale or with manual workarounds. The underlying code path has not changed.
### Observed behaviour
On a mainnet node restart (v3.4.0-dev, `bal-devnet-2` branch), 212 legacy **v1.0** snapshot files had no metadata available. The downloader resolved them from peers over ~10 minutes, with the rate declining as fewer peers had the data:
```
07:31:50 No metadata yet files=212
07:33:04 No metadata yet files=189 (~8/min)
07:35:34 No metadata yet files=121 (~23/min)
07:39:05 No metadata yet files=41 (~23/min)
07:41:50 No metadata yet files=15 (slowing to ~9/min)
07:42:32 OtterSync started (did not wait for metadata)
07:43:32 file-metadata=1081/1081 (all resolved)
```
In this instance, the node was configured with `snapCfg.Local=true`, so `SyncSnapshots` was skipped and execution proceeded independently. The no-metadata issue did **not** block startup.
However, if `snapCfg.Local=false` (the default for non-preverified setups), `SyncSnapshots` calls `DownloadSnapshots` → `batch.wait()`, which **would have blocked** until all 212 torrents completed.
The new trigger in this case is **legacy v1.0 format files** remaining on disk after a v1.0→v1.1 migration. The v1.1 webseed does not serve `.torrent` files for v1.0 snapshots, forcing metadata resolution to fall back to BitTorrent peer exchange — which is slow and unreliable as peers migrate away from v1.0.
### Root cause
The metadata resolution path in `addedFirstDownloader` (`db/downloader/downloader.go:1001`):
1. Tries webseed first: `fetchMetainfoFromWebseeds()` fetches `{webseed_url}/{name}.torrent`
2. If webseed fails (e.g. v1.0 files not served by the v1.1 webseed), falls back to BitTorrent peer exchange via `delayedGotInfoHandler()` → `<-t.GotInfo()`
3. `delayedGotInfoHandler` runs in a background goroutine — it does **not** block the batch setup
But `batch.wait()` in `db/downloader/download-batch.go:92` blocks on `t.Complete().On()` for every torrent:
```go
func (me *downloadBatch) wait(ctx context.Context) error {
defer me.abandon()
for _, t := range me.torrents {
select {
case <-t.Complete().On(): // blocks forever if no metadata
case <-t.Closed():
case <-ctx.Done(): // only fires on shutdown
}
}
return nil
}
```
A torrent without metadata cannot complete. `ctx.Done()` only fires on node shutdown (no timeout). This creates an indefinite hang.
### Conditions for hang
All of the following must be true:
1. `snapCfg.Local` is `false` (non-preverified snapshot configuration)
2. Snapshot files exist on disk without corresponding metadata (e.g. legacy v1.0 files after a format migration, or interrupted initial sync)
3. The webseed does not serve `.torrent` files for these snapshots
4. No BitTorrent peers are available with the metadata (or peer discovery is slow/failing)
### Impact
- Node startup hangs at the `SyncSnapshots` stage with no progress indication beyond the `No metadata yet` warnings
- Only recoverable by killing and restarting the node (and hoping peer conditions change), or manually removing the offending files
Contributor guide
Assessment
This issue has not been assessed yet.