ChainSafe / ChainSafe/lodestar

[fulu] Sync stuck due to peers returning 0 block for beacon_block_by_range request

Open
#8,147 9 comments 0 reactions 1 assignee Claimed by @twoeths View on GitHub
meta-bug scope-sync-updates
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

### Describe the bug

sync was stuck with this log

```
2025-08-08 11:07:09.405
Aug-08 04:07:09.405[sync] debug: Downloaded batch id=Finalized-0, startEpoch=8, status=AwaitingProcessing, blocks=0, peer=16...mAswsc

2025-08-08 11:07:09.177
Aug-08 04:07:09.176[sync] verbose: Downloading batch id=Finalized-0, startEpoch=8, status=AwaitingDownload, peer=16...mAswsc
2025-08-08 11:07:09.177
Aug-08 04:07:09.176[sync] verbose: Batch download error id=Finalized-0, startEpoch=8, status=Downloading, peer=16...aDqVMg, code=REQUEST_ERROR_TTFB_TIMEOUT
2025-08-08 11:07:04.143
Aug-08 04:07:04.143[sync] verbose: Downloading batch id=Finalized-0, startEpoch=8, status=AwaitingDownload, peer=16...aDqVMg
2025-08-08 11:07:04.143
Aug-08 04:07:04.143[sync] verbose: Advanced chain id=Finalized-0, lastEpochWithProcessBlocks=6
2025-08-08 11:07:03.947
Aug-08 04:07:03.947[sync] debug: Downloaded batch id=Finalized-0, startEpoch=5, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...dRN2bf
2025-08-08 11:07:03.941
Aug-08 04:07:03.941[sync] debug: Downloaded batch id=Finalized-0, startEpoch=6, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...dRN2bf
2025-08-08 11:07:03.581
Aug-08 04:07:03.581[sync] debug: Downloaded batch id=Finalized-0, startEpoch=7, status=AwaitingProcessing, blocks=0, peer=16...mAswsc
2025-08-08 11:07:03.344
Aug-08 04:07:03.343[sync] verbose: Downloading batch id=Finalized-0, startEpoch=5, status=AwaitingDownload, peer=16...dRN2bf
2025-08-08 11:07:03.344
Aug-08 04:07:03.343[sync] verbose: Downloading batch id=Finalized-0, startEpoch=7, status=AwaitingDownload, peer=16...mAswsc
2025-08-08 11:07:03.344
Aug-08 04:07:03.343[sync] verbose: Downloading batch id=Finalized-0, startEpoch=6, status=AwaitingDownload, peer=16...dRN2bf
2025-08-08 11:07:03.343
Aug-08 04:07:03.343[sync] verbose: Batch validation error id=Finalized-0, startEpoch=6, status=AwaitingValidation
2025-08-08 11:07:03.343
Aug-08 04:07:03.343[sync] verbose: Batch validation error id=Finalized-0, startEpoch=5, status=AwaitingValidation
2025-08-08 11:07:03.343
Aug-08 04:07:03.343[sync] verbose: Batch process error id=Finalized-0, startEpoch=7, status=Processing, code=BLOCK_ERROR_PARENT_UNKNOWN, parentRoot=0x907ff88e87331029dba28ecb50b0dcb8d073e51b069369dd2b357a9610cd3b1e
2025-08-08 11:07:03.308
Aug-08 04:07:03.308[sync] debug: Downloaded batch id=Finalized-0, startEpoch=7, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...dRN2bf
2025-08-08 11:07:02.660
Aug-08 04:07:02.660[sync] verbose: Downloading batch id=Finalized-0, startEpoch=7, status=AwaitingDownload, peer=16...dRN2bf
2025-08-08 11:07:02.660
Aug-08 04:07:02.659[sync] verbose: Batch download error id=Finalized-0, startEpoch=7, status=Downloading, peer=16...aDqVMg, code=REQUEST_ERROR_TTFB_TIMEOUT
2025-08-08 11:07:02.609
Aug-08 04:07:02.609[sync] debug: Downloaded batch id=Finalized-0, startEpoch=6, status=AwaitingProcessing, blocks=0, peer=16...mAswsc
2025-08-08 11:07:02.400
Aug-08 04:07:02.388[sync] verbose: Downloading batch id=Finalized-0, startEpoch=6, status=AwaitingDownload, peer=16...mAswsc
2025-08-08 11:07:02.400
Aug-08 04:07:02.388[sync] verbose: Batch download error id=Finalized-0, startEpoch=6, status=Downloading, peer=16...aDqVMg, code=REQUEST_ERROR_TTFB_TIMEOUT
2025-08-08 11:06:57.626
Aug-08 04:06:57.626[sync] verbose: Downloading batch id=Finalized-0, startEpoch=7, status=AwaitingDownload, peer=16...aDqVMg
2025-08-08 11:06:57.626
Aug-08 04:06:57.625[sync] verbose: Batch download error id=Finalized-0, startEpoch=7, status=Downloading, peer=16...fBGqqd, code=REQUEST_ERROR_RESP_TIMEOUT
2025-08-08 11:06:57.537
Aug-08 04:06:57.537[sync] verbose: Downloading batch id=Finalized-0, startEpoch=7, status=AwaitingDownload, peer=16...fBGqqd
2025-08-08 11:06:57.537
Aug-08 04:06:57.537[sync] verbose: Advanced chain id=Finalized-0, lastEpochWithProcessBlocks=5
2025-08-08 11:06:57.357
Aug-08 04:06:57.357[sync] verbose: Downloading batch id=Finalized-0, startEpoch=6, status=AwaitingDownload, peer=16...aDqVMg
2025-08-08 11:06:57.357
Aug-08 04:06:57.357[sync] verbose: Advanced chain id=Finalized-0, lastEpochWithProcessBlocks=4
2025-08-08 11:06:57.323
Aug-08 04:06:57.323[sync] debug: Downloaded batch id=Finalized-0, startEpoch=5, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...fBGqqd
2025-08-08 11:06:57.226
Aug-08 04:06:57.226[sync] verbose: Downloading batch id=Finalized-0, startEpoch=5, status=AwaitingDownload, peer=16...fBGqqd
2025-08-08 11:06:57.226
Aug-08 04:06:57.225[sync] verbose: Batch download error id=Finalized-0, startEpoch=5, status=Downloading, peer=16...aDqVMg, code=REQUEST_ERROR_TTFB_TIMEOUT
2025-08-08 11:06:57.146
Aug-08 04:06:57.146[sync] debug: Downloaded batch id=Finalized-0, startEpoch=4, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...fBGqqd
2025-08-08 11:06:57.017
Aug-08 04:06:57.016[sync] verbose: Downloading batch id=Finalized-0, startEpoch=4, status=AwaitingDownload, peer=16...fBGqqd
2025-08-08 11:06:57.017
Aug-08 04:06:57.016[sync] verbose: Batch download error id=Finalized-0, startEpoch=4, status=Downloading, peer=16...aDqVMg, code=REQUEST_ERROR_TTFB_TIMEOUT
2025-08-08 11:06:52.191
Aug-08 04:06:52.190[sync] verbose: Downloading batch id=Finalized-0, startEpoch=5, status=AwaitingDownload, peer=16...aDqVMg
2025-08-08 11:06:52.190
Aug-08 04:06:52.190[sync] verbose: Batch download error id=Finalized-0, startEpoch=5, status=Downloading, peer=16...fBGqqd, code=REQUEST_ERROR_RESP_TIMEOUT
2025-08-08 11:06:52.097
Aug-08 04:06:52.097[sync] verbose: Downloading batch id=Finalized-0, startEpoch=5, status=AwaitingDownload, peer=16...fBGqqd
2025-08-08 11:06:52.097
Aug-08 04:06:52.097[sync] verbose: Advanced chain id=Finalized-0, lastEpochWithProcessBlocks=3
2025-08-08 11:06:51.986
Aug-08 04:06:51.985[sync] verbose: Downloading batch id=Finalized-0, startEpoch=4, status=AwaitingDownload, peer=16...aDqVMg
2025-08-08 11:06:51.985
Aug-08 04:06:51.985[sync] verbose: Batch download error id=Finalized-0, startEpoch=4, status=Downloading, peer=16...fBGqqd, code=REQUEST_ERROR_RESP_TIMEOUT
2025-08-08 11:06:51.858
Aug-08 04:06:51.858[sync] verbose: Downloading batch id=Finalized-0, startEpoch=4, status=AwaitingDownload, peer=16...fBGqqd
2025-08-08 11:06:51.858
Aug-08 04:06:51.858[sync] verbose: Advanced chain id=Finalized-0, lastEpochWithProcessBlocks=2
2025-08-08 11:06:51.650
Aug-08 04:06:51.650[sync] debug: Downloaded batch id=Finalized-0, startEpoch=2, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...fBGqqd
2025-08-08 11:06:51.520
Aug-08 04:06:51.519[sync] verbose: Downloading batch id=Finalized-0, startEpoch=2, status=AwaitingDownload, peer=16...fBGqqd
2025-08-08 11:06:51.520
Aug-08 04:06:51.519[sync] verbose: Batch download error id=Finalized-0, startEpoch=2, status=Downloading, peer=16...aDqVMg, code=REQUEST_ERROR_TTFB_TIMEOUT
2025-08-08 11:06:50.420
Aug-08 04:06:50.420[sync] debug: Downloaded batch id=Finalized-0, startEpoch=3, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...fBGqqd
2025-08-08 11:06:50.292
Aug-08 04:06:50.292[sync] verbose: Downloading batch id=Finalized-0, startEpoch=3, status=AwaitingDownload, peer=16...fBGqqd
2025-08-08 11:06:50.292
Aug-08 04:06:50.292[sync] verbose: Advanced chain id=Finalized-0, lastEpochWithProcessBlocks=1
2025-08-08 11:06:49.856
Aug-08 04:06:49.855[sync] debug: Downloaded batch id=Finalized-0, startEpoch=0, status=AwaitingProcessing, blocks=31, blobs=0, dataColumns=0, peer=16...fBGqqd
2025-08-08 11:06:49.725
Aug-08 04:06:49.725[sync] verbose: Downloading batch id=Finalized-0, startEpoch=0, status=AwaitingDownload, peer=16...fBGqqd
2025-08-08 11:06:49.725
Aug-08 04:06:49.724[sync] verbose: Batch download error id=Finalized-0, startEpoch=0, status=Downloading, peer=16...aDqVMg, code=REQUEST_ERROR_TTFB_TIMEOUT
2025-08-08 11:06:46.488
Aug-08 04:06:46.488[sync] debug: Downloaded batch id=Finalized-0, startEpoch=1, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...fBGqqd
2025-08-08 11:06:46.486
Aug-08 04:06:46.486[sync] verbose: Downloading batch id=Finalized-0, startEpoch=2, status=AwaitingDownload, peer=16...aDqVMg
2025-08-08 11:06:46.357
Aug-08 04:06:46.357[sync] verbose: Downloading batch id=Finalized-0, startEpoch=1, status=AwaitingDownload, peer=16...fBGqqd
2025-08-08 11:06:44.685
Aug-08 04:06:44.685[sync] verbose: Downloading batch id=Finalized-0, startEpoch=0, status=AwaitingDownload, peer=16...aDqVMg
2025-08-08 10:58:16.750
Aug-08 03:58:16.750[sync] debug: Downloaded batch id=Finalized-0, startEpoch=4, status=AwaitingProcessing, blocks=0, peer=16...VbgfFx
2025-08-08 10:58:16.750
Aug-08 03:58:16.750[sync] debug: Downloaded batch id=Finalized-0, startEpoch=3, status=AwaitingProcessing, blocks=0, peer=16...VbgfFx
2025-08-08 10:58:16.697
Aug-08 03:58:16.697[sync] verbose: Downloading batch id=Finalized-0, startEpoch=4, status=AwaitingDownload, peer=16...VbgfFx
2025-08-08 10:58:16.697
Aug-08 03:58:16.697[sync] verbose: Downloading batch id=Finalized-0, startEpoch=3, status=AwaitingDownload, peer=16...VbgfFx
2025-08-08 10:58:16.697
Aug-08 03:58:16.697[sync] verbose: Batch download error id=Finalized-0, startEpoch=3, status=Downloading, peer=16...VbgfFx, code=REQUEST_ERROR_RESP_TIMEOUT
2025-08-08 10:58:16.627
Aug-08 03:58:16.627[sync] verbose: Advanced chain id=Finalized-0, lastEpochWithProcessBlocks=2
2025-08-08 10:58:16.613
Aug-08 03:58:16.613[sync] verbose: Downloading batch id=Finalized-0, startEpoch=3, status=AwaitingDownload, peer=16...VbgfFx
2025-08-08 10:58:16.613
Aug-08 03:58:16.613[sync] verbose: Batch download error id=Finalized-0, startEpoch=3, status=Downloading, peer=16...VbgfFx, code=REQUEST_ERROR_RESP_TIMEOUT
2025-08-08 10:58:16.527
Aug-08 03:58:16.527[sync] verbose: Downloading batch id=Finalized-0, startEpoch=3, status=AwaitingDownload, peer=16...VbgfFx
2025-08-08 10:58:16.527
Aug-08 03:58:16.527[sync] verbose: Batch download error id=Finalized-0, startEpoch=3, status=Downloading, peer=16...VbgfFx, code=REQUEST_ERROR_RESP_TIMEOUT
2025-08-08 10:58:16.435
Aug-08 03:58:16.435[sync] verbose: Downloading batch id=Finalized-0, startEpoch=3, status=AwaitingDownload, peer=16...VbgfFx
2025-08-08 10:58:16.435
Aug-08 03:58:16.435[sync] verbose: Advanced chain id=Finalized-0, lastEpochWithProcessBlocks=1
2025-08-08 10:58:16.110
Aug-08 03:58:16.110[sync] debug: Downloaded batch id=Finalized-0, startEpoch=2, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...VbgfFx
2025-08-08 10:58:16.105
Aug-08 03:58:16.105[sync] debug: Downloaded batch id=Finalized-0, startEpoch=1, status=AwaitingProcessing, blocks=32, blobs=0, dataColumns=0, peer=16...VbgfFx
2025-08-08 10:58:16.012
Aug-08 03:58:16.012[sync] verbose: Downloading batch id=Finalized-0, startEpoch=2, status=AwaitingDownload, peer=16...VbgfFx
2025-08-08 10:58:16.012
Aug-08 03:58:16.012[sync] verbose: Downloading batch id=Finalized-0, startEpoch=1, status=AwaitingDownload, peer=16...VbgfFx
2025-08-08 10:58:16.012
Aug-08 03:58:16.011[sync] debug: Downloaded batch id=Finalized-0, startEpoch=0, status=AwaitingProcessing, blocks=31, blobs=0, dataColumns=0, peer=16...VbgfFx
2025-08-08 10:58:15.879
Aug-08 03:58:15.879[sync] verbose: Downloading batch id=Finalized-0, startEpoch=0, status=AwaitingDownload, peer=16...VbgfFx
```

### Expected behavior

Sync should not stuck

### Steps to reproduce

_No response_

### Additional context

_No response_

### Operating system

Linux

### Lodestar version or commit hash

peerDAS

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.