hiero-ledger / hiero-ledger/hiero-consensus-node

Virtual map reconnect: improve virtual subtree snapshots on the teacher side

Open
#13,732 0 comments 0 reactions 0 assignees View on GitHub
P2 Performance Platform Platform Data Structures Platform Reconnect Platform Virtual Map Scalability
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

During reconnects, teachers make a snapshot for every virtual map in the state tree, before it starts sending it to learners. It starts with a call to `VirtualPipeline.detachCopy()` from `TeacherPullVirtualTreeView` or `TeacherPushVirtualTreeView` constructor and ends with `VirtualRootNode.detach()` This method does two things:

1. Creates a node cache snapshot
2. Creates a data source snapshot (using the current data source builder)

Node cache snapshot is more or less fine. It creates a new Java object and migrates only relevant mutations from the existing cache to it. Data source snapshot is something we can improve. Currently MerkleDb data source builder just calls `MerkleDbDataSource.snapshot()`, which hard-links all data files and dumps all indices to disk. Before such a snapshot can be used, all these indices must be loaded from disk, and it takes time.

This task is to create a similar method to `MerkleDbDataSource.snapshot()` that is only used by the teacher when teaching and skips the disk part (or add this to the existing method based on a flag or something).

For testing, the ReconnectBench is a good place to start and should show improvements if the implementation is faster. It is also easily profiled. There is an existing log statement that prints the duration of the `MerkleDbDataSource.snapshot()` method.

@jasperpotts suggests that we can avoid index serialization/deserialization in this case.

Contributor guide

Open the contributing guide

Research direction

Start at VirtualPipeline.detachCopy() in the TeacherPullVirtualTreeView and TeacherPushVirtualTreeView constructors, follow the flow through VirtualRootNode.detach(), and inspect MerkleDbDataSource.snapshot(). Run or profile ReconnectBench and use the existing snapshot-duration log as a baseline; done means teacher reconnect snapshots avoid unnecessary index disk work while preserving the required snapshot behavior and showing an improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.