ChainSafe / ChainSafe/lodestar

glamsterdam-devnet-5: sync stalled due to invalid payload cached forever

Open
#9,660 0 comments 0 reactions 0 assignees View on GitHub
meta-bug
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

### Describe the bug

## Incident (glamsterdam-devnet-7)

A peer served the slot-1312 builder envelope with `block_access_list` stripped to `0x` (canonical: 21,734 bytes) while keeping the original builder signature → `PAYLOAD_ERROR_INVALID_SIGNATURE`:

```
Block error slot=1312, code=BLOCK_ERROR_BEACON_CHAIN_ERROR, error=PAYLOAD_ERROR_INVALID_SIGNATURE
at BeaconChain.importExecutionPayload (importExecutionPayload.ts:192)
```

On `lodestar-nethermind-1` (v1.44.0):
- 73 batch failures over ~7h, all slot 1312, served by 13 distinct peers
- 96× `SeenPayloadEnvelopeInput.add reused existing entry` — every retry reused the cached bad envelope
- node state for 1312 was correct; the canonical envelope verifies fine against builder 0 — only the cached bytes were bad
- all 4 Lodestar nodes on the devnet were wedged the same way at different slots (1218, 1218, 1312, 1411)

### Expected behavior

An envelope failing verification during import is detached from the cache so the next batch retry attaches fresh bytes from another peer and sync self-heals.

### Steps to reproduce

_No response_

### Additional context

_No response_

### Operating system

Linux

### Lodestar version or commit hash

glamsterdam-devnet-7

Contributor guide

Open the contributing guide

Research direction

Start at importExecutionPayload.ts:192 and trace how the failing envelope is handled by SeenPayloadEnvelopeInput.add, especially the reused cached entry shown in the logs. Confirm that an envelope failing verification is detached so a retry can attach fresh bytes from another peer, and verify that sync self-heals for the reported stalled slots.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
blockchain, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.