ChainSafe / ChainSafe/lodestar
glamsterdam-devnet-5: sync stalled due to invalid payload cached forever
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 150
Description
### Describe the bug
## Incident (glamsterdam-devnet-7)
A peer served the slot-1312 builder envelope with `block_access_list` stripped to `0x` (canonical: 21,734 bytes) while keeping the original builder signature → `PAYLOAD_ERROR_INVALID_SIGNATURE`:
```
Block error slot=1312, code=BLOCK_ERROR_BEACON_CHAIN_ERROR, error=PAYLOAD_ERROR_INVALID_SIGNATURE
at BeaconChain.importExecutionPayload (importExecutionPayload.ts:192)
```
On `lodestar-nethermind-1` (v1.44.0):
- 73 batch failures over ~7h, all slot 1312, served by 13 distinct peers
- 96× `SeenPayloadEnvelopeInput.add reused existing entry` — every retry reused the cached bad envelope
- node state for 1312 was correct; the canonical envelope verifies fine against builder 0 — only the cached bytes were bad
- all 4 Lodestar nodes on the devnet were wedged the same way at different slots (1218, 1218, 1312, 1411)
### Expected behavior
An envelope failing verification during import is detached from the cache so the next batch retry attaches fresh bytes from another peer and sync self-heals.
### Steps to reproduce
_No response_
### Additional context
_No response_
### Operating system
Linux
### Lodestar version or commit hash
glamsterdam-devnet-7
Contributor guide
Research direction
Start at importExecutionPayload.ts:192 and trace how the failing envelope is handled by SeenPayloadEnvelopeInput.add, especially the reused cached entry shown in the logs. Confirm that an envelope failing verification is detached so a retry can attach fresh bytes from another peer, and verify that sync self-heals for the reported stalled slots.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- blockchain, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100