ChainSafe / ChainSafe/lodestar

Syncing strategy

Open
#8,225 2 comments 0 reactions 1 assignee Claimed by @nazarhussain View on GitHub
meta-discussion scope-performance scope-sync-updates
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 18h
Merged PRs (30d)
156

Description

Current syncing code is brittle and doesn't work well. The reason is that it has had features bolted on, and the sync strategy has not been updated as needed.

Below is a proposal for how to organize our syncing.

A first step is to list out some different situations the node may find itself in.
- The node has synced to a checkpoint in the past. We see peer statuses with a finalized chain ahead of our own.
- The node has synced to the latest finalized checkpoint. We see peer statuses with heads we haven't seen.
- The node has synced to the latest finalized checkpoint. The network has been experiencing a long period of non-finality and we see peer statuses with heads we haven't seen.
- The node has synced to head. We receive a gossip block with a parent we haven't seen.
- The node has synced to head. We receive a gossip block but haven't received all blobs / columns for that block.
- The node has synced to head. We receive a gossip blob/column but haven't received all blobs / columns or the block.

Currently, we have three modes of syncing (organized in two classes) to tackle these scenarios:
- Finalized range sync: This syncing mode organizes peers by status, uses the `by_range` requests to forward sync to a finalized checkpoint
- Unfinalized range sync: This syncing mode organizes peers by status, uses the `by_range` requests to forward sync.
- Unknown block sync: This syncing mode tracks unknown block roots and block inputs unavailable data and builds chains backwards and retrieves all blobs/columns.

This current strategy has several deficiencies:
- Unfinalized range sync has a race condition that cannot be avoided. Between statusing a peer and retrieving a batch of blocks, the peer's head may have changed. We cannot be sure which chain of blocks the peer will be sending us. Matters are made worse when considering blobs and columns. Syncing becomes unpredictable.
- Unknown block sync cannot handle long periods of non-finality. Because the syncing mechanism caches entire blocks + blobs/columns in memory, a strict cap must be enacted to avoid OOMing. Thus, we cannot rely on unknown block sync to handle periods of non-finality longer than this cap. We have to fall back to the unpredictable unfinalized range sync

We can overcome these deficiencies in the following way:
- Unfinalized range sync must use `by_root` requests to forward sync. This will ensure predictable syncing by always knowing which chain we are syncing. This implies that the header chain is already known in advance.
- Unknown block sync must decouple caching blocks + data from crafting a chain of headers. This will allow us to build long chains of headers that can happen during non-finality and not be limited by an artificially-low cap.

To a path forward:
- We can keep finalized range sync as it is, no problems.
- We need a new unfinalized forward sync that is fed a chain of headers (slot, blockRoot), downloads batches of block inputs, feeds into the block processor. (It can use a block input cache to avoid re-downloading previously-seen stuff)
- We need a new unknown block sync that is fed unknown blocks / block-roots and finds parents backwards until it intersects with our chain or is irrelevant. It can't cache the entire block + blobs/columns, it must only handle simplified block headers.

Integration:
- When we get a block or blob that is not fully available (but its parent is known), the slot+root can be fed to unfinalized forward sync.
- When we get a block or blob where the parent is not known, the header can be fed to the unknown block sync
- The block input cache could still maintain block inputs that are not fully available, or when the parent is not known.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.