erigontech / erigontech/silkworm

Very first sync from scratch History index causes HUGE etl files - limit batch ?

Open
#744 0 comments 0 reactions 0 assignees View on GitHub
research
Dominant language
C++
Stars
318
Forks
84
PR merge metrics
No merged PRs in 30d

Description

During very first sync from scratch the execution of Stage History Index causes a lot of etl files to be flushed into `etl` directory.
On ethereum mainnet (at block ~15.4M) I recorded up to 150GB of etl data which eventually translates into the same amount of space when loaded into db. This implies when the db is already large enough (before History Index db is 1.65TB) user needs at least 300GB extra space for the stage to complete. Eventually etl gets freed.

Worth to investigate if it's viable to run History Index in batches (pretty much like Execution) to limit the generation of huge temporary data and make it restartable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.