ChainSafe / ChainSafe/lodestar

Gossip message received to validation latency

Open
#9,942 2 comments 0 reactions 0 assignees View on GitHub
meta-bug
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

### Describe the bug

`recvToValLatency` is the root cause of majority of wrong head vote on mainnet (+ when gossip block received late into the slot)
- validate gossip block
```
2026-08-23 07:24:51.127
Aug-23 00:24:51.127[network] debug: Received gossip block slot=15051722, currentSlot=15051722, peerId=16Uiu2HAmDtzcwLDvCTyoZ7CAh1nMpC3BJUkjQRFkd4MsGSdpFK8R, delaySec=3.6410000324249268, recvToValLatency=0.48600006103515625

2026-08-23 07:33:02.940
Aug-23 00:33:02.939[network] debug: Received gossip block slot=15051763, currentSlot=15051763, peerId=16Uiu2HAkz8J7FpbMWAiy6Vq8S2UMNp7Sx62g3LsGLC6YGyNMjDX3, delaySec=3.2060000896453857, recvToValLatency=0.7329998016357422

Aug-23 00:54:27.149[network] debug: Received gossip block slot=15051870, currentSlot=15051870, peerId=16Uiu2HAmNFy1RPmyskFNu1TNJ3sADdCGMjNKWkA6YSautThVVrdi, delaySec=3.434000015258789, recvToValLatency=0.7149999141693115

```

- verify payload

```
2026-08-23 08:59:27.400 | Aug-23 01:59:27.400[chain] debug: Verified execution payload slot=15052195, recvToValLatency=0.7590000629425049, recvToValidation=1.1770000457763672, validationTime=0.4179999828338623
```

- validate gossip data column (fulu)

```
Aug-23 01:59:27.432[network] debug: Received gossip dataColumn slot=15052195, blockRoot=0x0996…7d61, timeCreatedSec=1787450366.11, expectedColumns=128, receivedColumns=122, currentSlot=15052195, peerId=16Uiu2HAmA18CwM2hv6xDJXykbCDrnPob3p6rVx6UWKm4wTYLTyYq, delaySec=3.299999952316284, secFromSlot=4.107000112533569, gossipSubnet=92, columnIndex=92, recvToValLatency=0.8070001602172852, recvToValidation=1.1319999694824219, validationTime=0.3249998092651367

```

### Expected behavior

- low latency

### Steps to reproduce

_No response_

### Additional context

this is not related to #9939 because it happens before `notifyNewPayload()` engine api

### Operating system

Linux

### Lodestar version or commit hash

unstable

Contributor guide

Open the contributing guide

Research direction

Start by tracing the gossip block, execution-payload, and data-column validation paths that emit the shown network and chain logs, keeping the notifyNewPayload engine API boundary in view. Use the supplied mainnet timings to profile recvToValLatency; done means identifying the source of the delay and reducing it, although no target latency is specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
blockchain, typescript
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.