ChainSafe / ChainSafe/lodestar
engine_newPayload dispatched ~300ms after block receipt delays head votes on heavy blocks
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 150
Description
## Summary
On heavy blocks (high gas + many blobs), Lodestar dispatches `engine_newPayload` to the EL only **~300 ms after receiving the gossip block** — after gossip validation and part of the state transition, rather than early on receipt. Because validators cannot attest to an optimistically-imported head (`notOnOptimisticBlockRoot`), **votability is gated on EL `VALID`**, so this late dispatch pushes EL `VALID` — and the head vote — past the 4 s attestation deadline. The EL itself is fast; the addressable delay is CL-side.
**This is not a one-off.** We operate a large fleet of Lodestar nodes on mainnet, and the pattern is **consistent across them**: **all of our Lodestar nodes miss the head vote on block slot 15051810** (misses are fully correlated per node — every validator on a node misses together)
## Environment
- mainnet, Lodestar CL + Nethermind EL
- Slot **15051810** / EL block **25814338** / `0xc8f361…339247`
- **13 blobs, 54.26M gas, 508 txs**, proposer 209291 (Titan builder)
- Slot start ≈ `2026-08-23 00:42:23.0 UTC`
## Logs (same host, clocks aligned)
**CL — Lodestar**
```
00:42:26.566 [network] debug: Received gossip block slot=15051810 delaySec=3.522 recvToValLatency=0.044
00:42:27.176 [chain] debug: Verified execution payload slot=15051810 recvToValLatency=0.045 recvToValidation=0.654 validationTime=0.609
00:42:27.224 [chain] verbose: Block processed slot=15051810 root=0xac4ec0… delaySec=4.224
```
**EL — Nethermind**
```
00:42:26.864 Received New Block: 25814338 (0xc8f361…)
00:42:26.869 Started pre-warming caches for block 25814338
00:42:27.141 Finished pre-warming caches for block 25814338
00:42:27.177 Valid. Result of New Block: 25814338
00:42:27.177 Processed 25814338 | 310.2 ms
00:42:27.273 Received ForkChoice: 25814338
```
## Timeline (seconds into the slot; slot start ≈ `00:42:23.0` UTC)
| into slot | where | event |
|---|---|---|
| 3.52 s | network thread | block received — gossipsub stamps `seenTimestampSec` (`delaySec=3.522`) |
| **3.56 s** | main thread | `Received gossip block` — handler starts (+44 ms bridge/pickup, `recvToValLatency=0.044`) |
| **3.86 s** | main thread → EL | `newPayload` dispatched; Nethermind receives it — **~298 ms of CL work** |
| 4.18 s | EL | Nethermind `Processed … 310.2 ms` → `VALID` |
| **4.22 s** | main thread | fork-choice import → `Block processed` (`delaySec=4.224`) |
**The 4 s attestation deadline falls between `newPayload` dispatch (3.86 s) and EL `VALID` (4.18 s)** — so the block is not votable at attestation time. Total receipt → import ≈ **0.70 s**, of which only **~0.31 s is the EL**; the block was already **3.52 s** into the slot on arrival, leaving essentially no margin.
## Analysis
1. **The block is valid and was available in time for the majority.** It became canonical with **60.8%** of the network voting it as head; **39.2% voted the parent** (a network-wide miss driven by the ~3.5 s arrival). This is not a bad/unusable block — it's a marginally-late block most of the network still handled. Lodestar's lateness is therefore a **CL-side processing issue**, not a block issue — and it's consistent enough that our entire Lodestar fleet lands on the losing side of these votes.
2. **The EL is not the bottleneck.** Nethermind executed 54.26M gas in **310 ms** (~175 Mgas/s) — healthy.
3. **The addressable delay is the ~298 ms CL-side window before `newPayload`.** The block was picked up 44 ms after arrival (`recvToValLatency=0.044`), yet a further **~298 ms** elapsed (3.56 s → 3.86 s) before the EL even received the `newPayload` call. Note `validationTime=0.609` on the CL is **not** the EL time (310 ms) — roughly half of it is CL-side.
4. **Votability is gated on EL `VALID`.** Validators must not attest to an optimistically-imported head, so the block only becomes votable once the EL returns `VALID` (~4.18 s here). Dispatching `newPayload` ~300 ms late directly pushes that past the 4 s deadline.
## Contributing factors to the ~298 ms (in order)
- **`newPayload` is dispatched late in the pipeline.** It fires after gossip validation + part of the state transition, instead of **early on receipt** so EL execution overlaps CL work. This is the primary lever. See #6381 (*Do early notifyNewPayload call to execution engine*).
- **Event-loop contention from concurrent data-column validation.** A 13-blob block → up to 128 column sidecars validated on the main thread around the same time, stretching the block's path to `newPayload`. Gossip-path column KZG is verified **one column at a time** (`validateGossipFuluDataColumnSidecar` → per-column `asyncVerifyCellKzgProofBatch`), unlike the **batched** reqresp path (`downloadByRange`/`downloadByRoot`).
- **JSON engine-API serialization scales with payload size.** `newPayload` sends the full `ExecutionPayload` as JSON — hex-encoding 508 txs (hex doubles the bytes) + allocation/GC. `jsonRpcHttpClient` hard-codes `Content-Type: application/json`. This grows with block/blob size and is what the SSZ engine API targets — Lodestar client-side impl in #9382 (open; spec `ethereum/execution-apis#764` → #793).
## Ask
- review **early `newPayload`** (#6381)
- **SSZ transport** (#9382) will likely help
Contributor guide
Research direction
Start with #6381 and trace the gossip block path around validateGossipFuluDataColumnSidecar, asyncVerifyCellKzgProofBatch, and the newPayload dispatch. Read jsonRpcHttpClient and compare the per-column gossip validation with downloadByRange/downloadByRoot, then use the reported timeline as the baseline. Done means identifying the CL-side delay and determining whether early newPayload or the transport work in #9382 addresses it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- blockchain, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100