hiero-ledger / hiero-ledger/hiero-consensus-node
Virtual map reconnect: improve virtual subtree snapshots on the teacher side
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
During reconnects, teachers make a snapshot for every virtual map in the state tree, before it starts sending it to learners. It starts with a call to `VirtualPipeline.detachCopy()` from `TeacherPullVirtualTreeView` or `TeacherPushVirtualTreeView` constructor and ends with `VirtualRootNode.detach()` This method does two things:
1. Creates a node cache snapshot
2. Creates a data source snapshot (using the current data source builder)
Node cache snapshot is more or less fine. It creates a new Java object and migrates only relevant mutations from the existing cache to it. Data source snapshot is something we can improve. Currently MerkleDb data source builder just calls `MerkleDbDataSource.snapshot()`, which hard-links all data files and dumps all indices to disk. Before such a snapshot can be used, all these indices must be loaded from disk, and it takes time.
This task is to create a similar method to `MerkleDbDataSource.snapshot()` that is only used by the teacher when teaching and skips the disk part (or add this to the existing method based on a flag or something).
For testing, the ReconnectBench is a good place to start and should show improvements if the implementation is faster. It is also easily profiled. There is an existing log statement that prints the duration of the `MerkleDbDataSource.snapshot()` method.
@jasperpotts suggests that we can avoid index serialization/deserialization in this case.
Contributor guide
Research direction
Start at VirtualPipeline.detachCopy() in the TeacherPullVirtualTreeView and TeacherPushVirtualTreeView constructors, follow the flow through VirtualRootNode.detach(), and inspect MerkleDbDataSource.snapshot(). Run or profile ReconnectBench and use the existing snapshot-duration log as a baseline; done means teacher reconnect snapshots avoid unnecessary index disk work while preserving the required snapshot behavior and showing an improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100