erigontech / erigontech/erigon

Increase merge limit of Blocks segments to 500K

Open
#15,821 3 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

Now blocks and bor files are 100K limited. It's because E2 was not able to release small files on R2. But now we can - so, we can afford bigger merge limit.

It will simplify code: now some chains has 500K limit for old files and 100K for new. (some users
It will improve performance: less shards (in sharded data) - better performance (less bloom filters checks, etc...). For example lookup txn by hash will be faster. (I saw this bottleneck in BSC pprof of trace_transaction)

Need do same for bor files (they are aligned with blocks)

This is not urgent task - let's work on it after 3.1 release.
Probably we don't need any special files change - all snapshotters and user's code will just do +1 merge in background and release naturally.

Also maybe it's good chance to change how `ExistenceFilter` works in recsplit index. I don't know if It make sense to do this features together or not (maybe not).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.