ChainSafe / ChainSafe/lodestar

Range sync bans peers when the execution client wrongly returns INVALID

Open
#10,031 4 comments 0 reactions 0 assignees View on GitHub
spec-gloas
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 18h
Merged PRs (30d)
156

Description

### Describe the bug

`EXECUTION_ENGINE_INVALID` is treated as a peer attributable failure in range sync (`isPeerAttributableFailure`). When a batch fails `MAX_BATCH_PROCESSING_ATTEMPTS` times with it, `shouldReportPeerOnBatchError` reports every peer that served a failed attempt with `LowToleranceError` (-10), so a peer is banned (-50) after 5 rounds.

This is correct when a peer serves a block with a bad payload, but the same verdict is produced when our own execution client is wrong. On glamsterdam-devnet-9 (2026-09-04, genesis resync) besu 26.9-develop marked a canonical payload `INVALID` after a "World State Root does not match" on block 4334 and kept its descendants in the bad block cache, returning `INVALID` without re-execution (reported to besu). Lodestar retried the batch, every retry was served by other peers with the correct blocks (same roots, hash verified) and every round penalized them.

Observed on the besu-2 node over ~2.7 h: 219 peer bans and 1140 batch processing errors with the head stuck at slot 4832. The four other nodes resyncing at the same time with the same peer set had 7 to 21 bans.

### Expected behavior

When independent peers serve identical data for a batch and the execution client still returns `INVALID`, the peers are not at fault and should not be penalized. `Batch.failedProcessingAttempts` already records the peers per attempt, and since #10025/#10028 the rejected block is dropped from the seen cache and re-downloaded, so the retries do compare independent peers.

Options

1. do not report peers for `EXECUTION_ENGINE_INVALID` once two or more distinct peers served the same block roots for the failing batch
2. or report `EXECUTION_ENGINE_INVALID` with `MidToleranceError` so it alone cannot ban a peer, and keep `LowToleranceError` for failures that prove bad content (`INVALID_SIGNATURE`, `INVALID_STATE_ROOT`, non-linear segments)

### Additional context

- #10029 surfaces the stall itself (warn after max attempts, `lodestar_sync_range_batch_process_error_total{code}`)
- #9877 / #9878: the ban wave has the same self sustaining shape, fewer peers concentrate the penalties on the remaining ones
- besu side: in-memory bad block manager, cleared by a restart, re-poisoned within minutes under CL driven genesis sync

Contributor guide

Open the contributing guide

Research direction

Start by tracing isPeerAttributableFailure and shouldReportPeerOnBatchError, then inspect Batch.failedProcessingAttempts and the retry path. Reproduce or add coverage for repeated EXECUTION_ENGINE_INVALID results after identical block roots are served by independent peers. Done means those peers are not incorrectly penalized while failures proving bad content still retain their existing penalties.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.