IntersectMBO / IntersectMBO/ouroboros-consensus

Increase sharing for in-memory headers

Open
#790 3 comments 0 reactions 0 assignees View on GitHub
🏎️ performance
Dominant language
Haskell
Stars
67
Forks
43
Avg merge
5d 13h
Merged PRs (30d)
43

Description

1. The ChainSync Client receives headers from the network layer. Currently, for each candidate, all headers are kept in memory. Even after input-output-hk/ouroboros-network#465 a large number of headers will be kept in memory.
2. The ChainDB will keep the headers of blocks in memory to perform chain selection. These headers are extracted from the blocks, which are read from disk.

Unfortunately, each header is kept in memory *twice*, because they are deserialised twice, so they won't be pointers to the same chunk of memory. If each header is 1kB and there are ~2k headers for each node in memory, we have 2x ~2MB of headers *per node*.

By sharing these headers between the ChainSync Client and the ChainDB, we can halve this. For example, by "interning" headers using a header table, a map from point to header.

Contributor guide

Open the contributing guide

Research direction

Trace how the ChainSync Client receives candidate headers and how the ChainDB extracts headers from blocks. Study the proposed header table mapping points to headers, then determine how both components can share one deserialized representation. Done means the duplicate in-memory headers are eliminated without changing chain selection or synchronization behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
blockchain, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.