erigontech / erigontech/erigon

Merge: impacting chain-tip

Open
#20,638 2 comments 0 reactions 1 assignee Claimed by @AskAlexSharov View on GitHub
ErigonDB performance
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

Problems:
- commitment.kv merge producing much storage.kv refferences read. Need try move to MMAP with `MADV_WILLNEED`
- Bloomfilters are stored in RAM (instead mmap). And during merge we can store 2x of Bllomfilters. Such "short peak memory jump" impacting ChainTip
- `Recsplit.build() -> buildWithWorkers -> GolombRice.Append` stored in RAM. Need move to Bufio or MMAP. (2g on some bloatnet file). Such "short peak memory jump" impacting ChainTip. Bloatnet got OOM because of this https://github.com/erigontech/erigon/issues/20560
- `Recsplit.build() -> buildOffsetsEf -> NewEliasFano -> EliasFano.deriveFields` storing in RAM. Need move to Bufio or MMAP. (4g on some bloatnet file). Such "short peak memory jump" impacting ChainTip. Bloatnet got OOM because of this https://github.com/erigontech/erigon/issues/20560
- `.ef` files merge have several in-mem buffers/arrays. For example one of them: https://github.com/erigontech/erigon/pull/20597
- domain.collate: double-buffering remove: https://github.com/erigontech/erigon/pull/20742
- post mortem on 32G validator: https://github.com/erigontech/erigon/issues/18483
- Commitment.Domain deref: released files on `main` (for 3.6)
- BTindex.Build(): greedy EliadFano https://github.com/erigontech/erigon/pull/21777
- BTindex.Build(): greedy keys copy

FYI: recent Merge-related PR's https://github.com/erigontech/erigon/issues/19430

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.