hiero-ledger / hiero-ledger/hiero-consensus-node

MerkleDb: reuse the same file for multiple flushes

Open
#8,795 0 comments 0 reactions 0 assignees View on GitHub
Performance Platform
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

### Problem

Currently, we create one new file during each flush for each of the three stores (path to KV, path to hash, key to path). Index is updated on the fly so the new file is accessible concurrently. We might keep the file open between flushes and close it only when it reaches some threshold, say 10g. This way we can avoid one compaction of files currently created by individual flushes.

Considerations:
* during a snapshot we must close all current flush write files to allow creating hard links to immutable files in the database;
* a file now may contain multiple instances of the same key; whenever we need to determine the latest valid key entry we need to take file offset into consideration (the largest offset is the most recent); that affects the state validation tool and index recovery;
* we may consider doing flushes more often as now we should not be concerned about creating too many small files; that may decrease the amount of data we keep in `VirtualNodeCache`.

### Solution

See above.

### Alternatives

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing MerkleDb flushes, snapshots, index recovery, and the state validation tool. Define how write files remain open across flushes, close for snapshots, and resolve duplicate keys by the largest file offset. Done means the three stores reuse files safely and the affected recovery and validation behavior is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.