hiero-ledger / hiero-ledger/hiero-consensus-node
Improved State Saving
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
### Problem
There are a number of things we can do to make State Saving lighter weight for Node effort, disk space and network upload cost.
The background is every node creates a saved state every 15min that gets uploaded to backup buckets. The saved state is used for two purposes:
- Node Restart - this has requirements to be as fast as possible to load
- Backups - this wants to be as light as possible, to minimize effort spent in CPU, disk and network
Today one code path is used for both and write all data to disk,
- Serialized Merkle Nodes
- Virtual Map DBs
- Database file hard links, hashes, leaves and half disk hash maps
- In memory index dumps
All of these files get uploaded. There is more saved and uploaded than needs to be.
### Solution
Reduce what is saved is saved states for backup while still keeping enough saved for restarts to be fast.
- Fix database to rebuild index and hashes if missing
- Change uploading to not upload hashes and in memory indexes. Maybe can be done by filter file names to be uploaded. This will depend on https://github.com/hashgraph/hedera-services/issues/5180 for leaf hashes being in separate files with internal hashes. This will be problematic for state validation tool. Could fix by uploading a hash of indexes rather than indexes them self.
- Optimize upload to not upload DB files that have been previously uploaded. Because DB files are immutable then if we have uploaded map X file 17 then we do not need to upload file 17 over and over. We will ideally need a server side improvement for this to create symbolic links for files not copied. Maybe that could be done with uploading tar files or some other container that can carry links.
- Optimize DB Compaction and VirtualNodeCache Flushing to minimize file churn, at the moment all data is rewritten at least once a day. Also we flush to many small files that get rolled up multiple times a day. This can be tuned with the aim of minimizing the new data files created that need to be uploaded.
- Compress uploads, there is a substantial file size saving of between 2x and 10x to be gained on saved states by compressing them with Zstandard compression. Ideally as we stream from node to backup server we will stream compress so the resulting files are compressed on wire and in storage at remote end. Zstandard can be more efficient CPU wise if we pre-build a dictionary, so because each set of files contains similar data we can train compression dictionaries for each set of files and use it for all new files in that set with high efficiency.
- Maybe? Optimize state saving to only write indexes to disk when doing a freeze for restart/upgrade if the cost of extra time during disaster recovery is acceptable.
Having partial data in saved states will reduce storage and upload costs substantially but increase risk of bad data. To mitigate that risk we can depend on the State Validation Tool to validate each uploaded state is complete and hashes correctly.
```[tasklist]
### Tasks
- [ ] https://github.com/hashgraph/hedera-services/issues/7396
- [ ] https://github.com/hashgraph/hedera-services/issues/7540
- [ ] https://github.com/hashgraph/hedera-services/issues/7502
- [ ] https://github.com/hashgraph/hedera-services/issues/7529
- [ ] https://github.com/hashgraph/hedera-services/issues/7525
- [ ] https://github.com/hashgraph/hedera-services/issues/7501
- [ ] https://github.com/hashgraph/hedera-services/issues/7499
- [ ] https://github.com/hashgraph/hedera-services/issues/7498
- [ ] https://github.com/hashgraph/hedera-services/issues/8795
- [ ] https://github.com/hashgraph/hedera-services/issues/9825
```
Contributor guide
Assessment
This issue has not been assessed yet.