hiero-ledger / hiero-ledger/hiero-consensus-node

Support MerkleDb snapshot archives

Open
#9,138 0 comments 0 reactions 0 assignees View on GitHub
Improvement P3 Platform Platform Data Structures Platform Virtual Map
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

MerkleDb snapshots contain the following files/stores:

* internalHashStoreDisk/ - a snapshot of all hashes on disk
* pathToDiskLocationInternalNodes.ll - an index for the store above
* internalHashStoreRam.hl - if hashes RAM/disk threshold is greater than 0, a snapshot of all hashes in RAM
* objectKeyToPath/ - key to path mappings, for generic (not long) keys
* pathToHashKeyValue/ - a snapshot of all key/value pairs
* pathToDiskLocationLeafNodes.ll - an index for the store above

These files are all required for fast node startups from saved state snapshots. However, snapshots are not only used for DR needs (to start a node), but also as long-term backups and to bootstrap mirror nodes. In this case, many files/stores in the list are redundant:

* internalHashStoreDisk and its index: all hashes can be restored by hashing the whole virtual tree
* same for internalHashStoreRam
* objectKeyToPath: iterate over the whole virtual tree and reconstruct this mapping
* pathToDiskLocationLeafNodes index: iterate over all store data files, for every path find the latest file/offset and use it for the index

The only really essential part of the snapshot is pathToHashKeyValue. Using this store, everything else can be restored. It will take quite some time, but this is acceptable for such archived snapshot.

Today we have code to restore some of the indices. It's a part of MerkleDb runtime (when a store is loaded, it checks for the corresponding index file and restores it if missing), but for performance reasons it is never used in production. There is no code to restore hashes or object key to path mapping. This ticket is to implement them.

An open question is whether it should be a standalone tool, somewhat similar to state validation tool, or it should be a built in to MerkleDb. I tend to think it should be the former.

Contributor guide

Open the contributing guide

Research direction

Start with the MerkleDb runtime's existing index-restoration path and compare it with the state validation tool mentioned in the issue. Determine whether archived snapshot restoration should be a standalone tool or built into MerkleDb. Done means restoring the hash stores and objectKeyToPath mapping from pathToHashKeyValue, while retaining the existing index restoration behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.