erigontech / erigontech/erigon
Merge: impacting chain-tip
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
Problems:
- commitment.kv merge producing much storage.kv refferences read. Need try move to MMAP with `MADV_WILLNEED`
- Bloomfilters are stored in RAM (instead mmap). And during merge we can store 2x of Bllomfilters. Such "short peak memory jump" impacting ChainTip
- `Recsplit.build() -> buildWithWorkers -> GolombRice.Append` stored in RAM. Need move to Bufio or MMAP. (2g on some bloatnet file). Such "short peak memory jump" impacting ChainTip. Bloatnet got OOM because of this https://github.com/erigontech/erigon/issues/20560
- `Recsplit.build() -> buildOffsetsEf -> NewEliasFano -> EliasFano.deriveFields` storing in RAM. Need move to Bufio or MMAP. (4g on some bloatnet file). Such "short peak memory jump" impacting ChainTip. Bloatnet got OOM because of this https://github.com/erigontech/erigon/issues/20560
- `.ef` files merge have several in-mem buffers/arrays. For example one of them: https://github.com/erigontech/erigon/pull/20597
- domain.collate: double-buffering remove: https://github.com/erigontech/erigon/pull/20742
- post mortem on 32G validator: https://github.com/erigontech/erigon/issues/18483
- Commitment.Domain deref: released files on `main` (for 3.6)
- BTindex.Build(): greedy EliadFano https://github.com/erigontech/erigon/pull/21777
- BTindex.Build(): greedy keys copy
FYI: recent Merge-related PR's https://github.com/erigontech/erigon/issues/19430
Contributor guide
Assessment
This issue has not been assessed yet.