ChainSafe / ChainSafe/lodestar
Range sync bans peers when the execution client wrongly returns INVALID
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 156
Description
### Describe the bug
`EXECUTION_ENGINE_INVALID` is treated as a peer attributable failure in range sync (`isPeerAttributableFailure`). When a batch fails `MAX_BATCH_PROCESSING_ATTEMPTS` times with it, `shouldReportPeerOnBatchError` reports every peer that served a failed attempt with `LowToleranceError` (-10), so a peer is banned (-50) after 5 rounds.
This is correct when a peer serves a block with a bad payload, but the same verdict is produced when our own execution client is wrong. On glamsterdam-devnet-9 (2026-09-04, genesis resync) besu 26.9-develop marked a canonical payload `INVALID` after a "World State Root does not match" on block 4334 and kept its descendants in the bad block cache, returning `INVALID` without re-execution (reported to besu). Lodestar retried the batch, every retry was served by other peers with the correct blocks (same roots, hash verified) and every round penalized them.
Observed on the besu-2 node over ~2.7 h: 219 peer bans and 1140 batch processing errors with the head stuck at slot 4832. The four other nodes resyncing at the same time with the same peer set had 7 to 21 bans.
### Expected behavior
When independent peers serve identical data for a batch and the execution client still returns `INVALID`, the peers are not at fault and should not be penalized. `Batch.failedProcessingAttempts` already records the peers per attempt, and since #10025/#10028 the rejected block is dropped from the seen cache and re-downloaded, so the retries do compare independent peers.
Options
1. do not report peers for `EXECUTION_ENGINE_INVALID` once two or more distinct peers served the same block roots for the failing batch
2. or report `EXECUTION_ENGINE_INVALID` with `MidToleranceError` so it alone cannot ban a peer, and keep `LowToleranceError` for failures that prove bad content (`INVALID_SIGNATURE`, `INVALID_STATE_ROOT`, non-linear segments)
### Additional context
- #10029 surfaces the stall itself (warn after max attempts, `lodestar_sync_range_batch_process_error_total{code}`)
- #9877 / #9878: the ban wave has the same self sustaining shape, fewer peers concentrate the penalties on the remaining ones
- besu side: in-memory bad block manager, cleared by a restart, re-poisoned within minutes under CL driven genesis sync
Contributor guide
Research direction
Start by tracing isPeerAttributableFailure and shouldReportPeerOnBatchError, then inspect Batch.failedProcessingAttempts and the retry path. Reproduce or add coverage for repeated EXECUTION_ENGINE_INVALID results after identical block roots are served by independent peers. Done means those peers are not incorrectly penalized while failures proving bad content still retain their existing penalties.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100