hiero-ledger / hiero-ledger/hiero-consensus-node
AbstractLongList.writeToFile clogs the pipeline
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
### Problem
When HalfDiskHashMap takes a snapshot it records the entire `bucketIndexToBucketLocation` to a file.
With a large number of buckets, it may take significant time to finish writing. Flushing is stopped during taking a snapshot and may accumulate enough data to trigger back-pressure that affects transaction handling.
Observed at `maxNumOfKeys` = 10B and TPS > 10K.
### Solution
Possible solutions:
* map the target file in memory, ensure actual recording is done asynchronously
* make a copy of the index to quickly release the pipeline for flushing
### Relevant Issues:
#8841 Need a way to change the number of object key buckets in MerkleDb
Contributor guide
Research direction
Start with AbstractLongList.writeToFile and trace how HalfDiskHashMap records bucketIndexToBucketLocation during snapshots. Read the snapshot and flushing paths, then determine how the chosen approach should prevent snapshot recording from blocking flushing while preserving the index data. Done means large snapshots no longer create the reported pipeline back-pressure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100