hiero-ledger / hiero-ledger/hiero-consensus-node

Fast MerkleDb snapshots with no index files

Open
#25,822 1 comment 0 reactions 1 assignee Claimed by @artemananiev View on GitHub
Platform Platform Data Structures
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

This is a part of ZDT epic: https://github.com/hiero-ledger/hiero-consensus-node/issues/25820

MerkleDb snapshots are relatively fast, but not as fast as needed for ZDT upgrades. With 1B states, database indices may be more than 50Gb in RAM. To write them to disk may take minutes (even with fast local NVMe drives), which is completely unacceptable.

Instead, the idea is to have indices shared between two CN processes, so they don't need to be written / read. Also, there should be a way to make a "fast" snapshot, which only includes data files - they are really fast, since no files are copied, only hard-links created.

Purely theoretical: such fast and lightweight snapshots can still be used outside of ZDT process. Database indexes can be restored by traversing all data files, this is long, but doable. This is not a goal for ZDT, though.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.