hiero-ledger / hiero-ledger/hiero-consensus-node
MerkleDb: reuse the same file for multiple flushes
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
### Problem
Currently, we create one new file during each flush for each of the three stores (path to KV, path to hash, key to path). Index is updated on the fly so the new file is accessible concurrently. We might keep the file open between flushes and close it only when it reaches some threshold, say 10g. This way we can avoid one compaction of files currently created by individual flushes.
Considerations:
* during a snapshot we must close all current flush write files to allow creating hard links to immutable files in the database;
* a file now may contain multiple instances of the same key; whenever we need to determine the latest valid key entry we need to take file offset into consideration (the largest offset is the most recent); that affects the state validation tool and index recovery;
* we may consider doing flushes more often as now we should not be concerned about creating too many small files; that may decrease the amount of data we keep in `VirtualNodeCache`.
### Solution
See above.
### Alternatives
_No response_
Contributor guide
Research direction
Start by tracing MerkleDb flushes, snapshots, index recovery, and the state validation tool. Define how write files remain open across flushes, close for snapshots, and resolve duplicate keys by the largest file offset. Done means the three stores reuse files safely and the affected recovery and validation behavior is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100