ChainSafe / ChainSafe/lodestar
Range sync rate limit exhaust when execution node is syncing
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 150
Description
### Describe the bug
When beacon node start and tries to sync from genesis, it usually fall back to range sync because head of connected peers are too far in the future.
1. It tries to call `BeaconBlocksByRange` method to get blocks from connected peer.
2. If the execution engine in the meanwhile is syncing it failed to process the blocks with
```
Batch process error id=Finalized, startEpoch=4, status=Processing, code=BLOCK_ERROR_EXECUTION_ERROR, execStatus=UNSAFE_OPTIMISTIC_STATUS, errorMessage=not safe to import SYNCING payload within 128 of currentSlot
```
3. It does not stop with this error and keep calling `BeaconBlocksByRange` continuously.
4. This exhaust the rate limit for `BeaconBlocksByRange` which is 1024 blocks per 10s.
https://github.com/ChainSafe/lodestar/blob/5047410270ed7d74f686d6fab3e9d0217a6bcf6a/packages/beacon-node/src/network/reqresp/rateLimit.ts#L31
5. Till the connected execution engine sync, beacon node can't proceed sync further because of rate limit on `BeaconBlocksByRange`.
### Expected behavior
It should be able to proceed with range sync after waiting the execution node to be synced.
**Possible solution could be to wait for the execution engine to sync before initiating multiple concurrent `BeaconBlocksByRange` requests.**
### Steps to reproduce
1. Run a node pair and wait for couple of epochs
2. Run the second node pair in sequence that beacon node and execution node starts same time.
### Additional context
Look for the logs of `range-sync-node-beacon-lodestar.log`
[multi-fork.zip](https://github.com/ChainSafe/lodestar/files/14308957/multi-fork.4.zip)
### Operating system
Linux
### Lodestar version or commit hash
unstable
Contributor guide
Assessment
This issue has not been assessed yet.