ChainSafe / ChainSafe/lodestar
Gloas payload-by-root sync retries forever when peer selection comes up empty
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 156
Description
Observed on a local gloas kurtosis devnet while e2e testing #9832: a node missing the payload envelope for an early slot never recovered it and stayed on its own fork, retrying the same download for the rest of the run with an ever-growing delay
```
Aug-15 18:44:55.082[sync] debug: Ignoring unknown payload root after failed download slot=1, root=0x9b7c...dc29, pendingPayloads=1, delaySec=48.06 - Error fetching payload by root slot=1 root=0x9b7c...dc29 after 2: cannot find peer with needed columns=[]
```
`pendingColumns` was empty and `filterPeers` treats an empty set as matching any peer, so the balancer's peer list must have been empty or every peer was excluded, rate limited or at its concurrent request cap. The node was otherwise participating in the network, so it had connected peers.
Two things worth looking at:
- why peer selection comes up empty while peers are connected, `peersMeta` may not track the peers that could serve the request
- the retry loop never recovered even over minutes, an eventual attempt against fresh peers should succeed
Contributor guide
Research direction
Trace payload-by-root sync through the retry loop, then inspect pendingColumns, filterPeers, and peersMeta where peer selection returns no candidates. Reproduce the empty-selection case on the Gloas Kurtosis devnet or with the existing sync flow, and verify that a node retries with fresh eligible peers and eventually recovers the missing payload envelope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100