ChainSafe / ChainSafe/lodestar

Gloas payload-by-root sync retries forever when peer selection comes up empty

Open
#9,921 6 comments 0 reactions 0 assignees View on GitHub
spec-gloas
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 18h
Merged PRs (30d)
156

Description

Observed on a local gloas kurtosis devnet while e2e testing #9832: a node missing the payload envelope for an early slot never recovered it and stayed on its own fork, retrying the same download for the rest of the run with an ever-growing delay

```
Aug-15 18:44:55.082[sync] debug: Ignoring unknown payload root after failed download slot=1, root=0x9b7c...dc29, pendingPayloads=1, delaySec=48.06 - Error fetching payload by root slot=1 root=0x9b7c...dc29 after 2: cannot find peer with needed columns=[]
```

`pendingColumns` was empty and `filterPeers` treats an empty set as matching any peer, so the balancer's peer list must have been empty or every peer was excluded, rate limited or at its concurrent request cap. The node was otherwise participating in the network, so it had connected peers.

Two things worth looking at:

- why peer selection comes up empty while peers are connected, `peersMeta` may not track the peers that could serve the request
- the retry loop never recovered even over minutes, an eventual attempt against fresh peers should succeed

Contributor guide

Open the contributing guide

Research direction

Trace payload-by-root sync through the retry loop, then inspect pendingColumns, filterPeers, and peersMeta where peer selection returns no candidates. Reproduce the empty-selection case on the Gloas Kurtosis devnet or with the existing sync flow, and verify that a node retries with fresh eligible peers and eventually recovers the missing payload envelope.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.