IntersectMBO / IntersectMBO/ouroboros-consensus
Increase sharing for in-memory headers
- Dominant language
- Haskell
- Stars
- 67
- Forks
- 43
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 43
Description
1. The ChainSync Client receives headers from the network layer. Currently, for each candidate, all headers are kept in memory. Even after input-output-hk/ouroboros-network#465 a large number of headers will be kept in memory.
2. The ChainDB will keep the headers of blocks in memory to perform chain selection. These headers are extracted from the blocks, which are read from disk.
Unfortunately, each header is kept in memory *twice*, because they are deserialised twice, so they won't be pointers to the same chunk of memory. If each header is 1kB and there are ~2k headers for each node in memory, we have 2x ~2MB of headers *per node*.
By sharing these headers between the ChainSync Client and the ChainDB, we can halve this. For example, by "interning" headers using a header table, a map from point to header.
Contributor guide
Research direction
Trace how the ChainSync Client receives candidate headers and how the ChainDB extracts headers from blocks. Study the proposed header table mapping points to headers, then determine how both components can share one deserialized representation. Done means the duplicate in-memory headers are eliminated without changing chain selection or synchronization behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- blockchain, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100