erigontech / erigontech/erigon

Downloader: missing torrent metadata for old files can block node startup, or cause confusing logging

Open
#19,545 2 comments 0 reactions 1 assignee Claimed by @anacrolix View on GitHub
Networking
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

## Downloader: missing torrent metadata can block node startup indefinitely, or cuase confusing logging

The issue below was observred while testing and happened several times, while benign from a functional perspective (the node was configured with `snapCfg.Local=true`) in this scenario it is confusing becuase its not clear wether to abandon the run or simply wait.

This gets exacerbated with agentic testing as the agent incorectly assumes the process is not working and attempts to fix the problem. Which wastes agent time. I think the quickets fix is to not produce these logs if we're operating in `snapCfg.Local=true`. But I also think the commentry around v1.0 files is worth fixing with something more than a work around.

`snapCfg.Local=true`

When the downloader encounters snapshot files on disk that lack torrent metadata (`.torrent` files or cached metainfo), the `batch.wait()` function blocks until every torrent in the batch completes. A torrent without metadata can never determine what pieces it needs and therefore can never complete. The only escape is context cancellation, but the sync context has no timeout — it only cancels on shutdown. This means the node can hang indefinitely on startup.

This is a long-standing issue that has been reported multiple times over the years but has never been fixed at the code level:

- #4034 — "Erigon stuck in 'Waiting for torrents metadata' phase" (May 2022, workaround: `rm *.torrent` + restart)
- #4154 — "Stuck at 'Waiting for torrents metadata: 0/87'" (2022)
- #5042 — "Erigon must self-heal from 'Waiting for torrents metadata: 121/122' state" (Aug 2022, closed as stale)
- #7302 — "[1/15 Snapshots] Waiting for torrents metadata" (2023)
- #8861 — "[Sepolia] Stuck syncing with 'Waiting for torrents metadata: 38/39'" (2023)

All prior reports were closed either as stale or with manual workarounds. The underlying code path has not changed.

### Observed behaviour

On a mainnet node restart (v3.4.0-dev, `bal-devnet-2` branch), 212 legacy **v1.0** snapshot files had no metadata available. The downloader resolved them from peers over ~10 minutes, with the rate declining as fewer peers had the data:

```
07:31:50 No metadata yet files=212
07:33:04 No metadata yet files=189 (~8/min)
07:35:34 No metadata yet files=121 (~23/min)
07:39:05 No metadata yet files=41 (~23/min)
07:41:50 No metadata yet files=15 (slowing to ~9/min)
07:42:32 OtterSync started (did not wait for metadata)
07:43:32 file-metadata=1081/1081 (all resolved)
```

In this instance, the node was configured with `snapCfg.Local=true`, so `SyncSnapshots` was skipped and execution proceeded independently. The no-metadata issue did **not** block startup.

However, if `snapCfg.Local=false` (the default for non-preverified setups), `SyncSnapshots` calls `DownloadSnapshots` → `batch.wait()`, which **would have blocked** until all 212 torrents completed.

The new trigger in this case is **legacy v1.0 format files** remaining on disk after a v1.0→v1.1 migration. The v1.1 webseed does not serve `.torrent` files for v1.0 snapshots, forcing metadata resolution to fall back to BitTorrent peer exchange — which is slow and unreliable as peers migrate away from v1.0.

### Root cause

The metadata resolution path in `addedFirstDownloader` (`db/downloader/downloader.go:1001`):

1. Tries webseed first: `fetchMetainfoFromWebseeds()` fetches `{webseed_url}/{name}.torrent`
2. If webseed fails (e.g. v1.0 files not served by the v1.1 webseed), falls back to BitTorrent peer exchange via `delayedGotInfoHandler()` → `<-t.GotInfo()`
3. `delayedGotInfoHandler` runs in a background goroutine — it does **not** block the batch setup

But `batch.wait()` in `db/downloader/download-batch.go:92` blocks on `t.Complete().On()` for every torrent:

```go
func (me *downloadBatch) wait(ctx context.Context) error {
defer me.abandon()
for _, t := range me.torrents {
select {
case <-t.Complete().On(): // blocks forever if no metadata
case <-t.Closed():
case <-ctx.Done(): // only fires on shutdown
}
}
return nil
}
```

A torrent without metadata cannot complete. `ctx.Done()` only fires on node shutdown (no timeout). This creates an indefinite hang.

### Conditions for hang

All of the following must be true:
1. `snapCfg.Local` is `false` (non-preverified snapshot configuration)
2. Snapshot files exist on disk without corresponding metadata (e.g. legacy v1.0 files after a format migration, or interrupted initial sync)
3. The webseed does not serve `.torrent` files for these snapshots
4. No BitTorrent peers are available with the metadata (or peer discovery is slow/failing)

### Impact

- Node startup hangs at the `SyncSnapshots` stage with no progress indication beyond the `No metadata yet` warnings
- Only recoverable by killing and restarting the node (and hoping peer conditions change), or manually removing the offending files

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.