ChainSafe / ChainSafe/lodestar

Range sync rate limit exhaust when execution node is syncing

Open
#6,435 0 comments 0 reactions 0 assignees View on GitHub
meta-bug meta-discussion
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

### Describe the bug

When beacon node start and tries to sync from genesis, it usually fall back to range sync because head of connected peers are too far in the future.

1. It tries to call `BeaconBlocksByRange` method to get blocks from connected peer.
2. If the execution engine in the meanwhile is syncing it failed to process the blocks with
```
Batch process error id=Finalized, startEpoch=4, status=Processing, code=BLOCK_ERROR_EXECUTION_ERROR, execStatus=UNSAFE_OPTIMISTIC_STATUS, errorMessage=not safe to import SYNCING payload within 128 of currentSlot
```
3. It does not stop with this error and keep calling `BeaconBlocksByRange` continuously.
4. This exhaust the rate limit for `BeaconBlocksByRange` which is 1024 blocks per 10s.
https://github.com/ChainSafe/lodestar/blob/5047410270ed7d74f686d6fab3e9d0217a6bcf6a/packages/beacon-node/src/network/reqresp/rateLimit.ts#L31
5. Till the connected execution engine sync, beacon node can't proceed sync further because of rate limit on `BeaconBlocksByRange`.

### Expected behavior

It should be able to proceed with range sync after waiting the execution node to be synced.

**Possible solution could be to wait for the execution engine to sync before initiating multiple concurrent `BeaconBlocksByRange` requests.**

### Steps to reproduce

1. Run a node pair and wait for couple of epochs
2. Run the second node pair in sequence that beacon node and execution node starts same time.

### Additional context

Look for the logs of `range-sync-node-beacon-lodestar.log`

[multi-fork.zip](https://github.com/ChainSafe/lodestar/files/14308957/multi-fork.4.zip)

### Operating system

Linux

### Lodestar version or commit hash

unstable

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.