hiero-ledger / hiero-ledger/hiero-consensus-node
Tweak sync permit return condition
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
## Background
- With the health monitor, nodes under high stress have sync permits revoked if any queue is too full
- Testing has shown that nodes with many revoked permits have a duplicate event rate decrease (~30% instead of ~60%)
## Problem
- During periods of stress, we should ideally reduce the duplicate event rate even more, down close to 0%
- The reason I think that we still see something like 30% duplicate rate, even for nodes that have only 1 sync permit, is that sync x completes, the permit is returned, and then sync x+1 begins before the events from sync x have been handled.
## Proposal
- We could reduce the duplicate event rate for nodes under heavy stress if we simply wait to return sync permits until all events from that sync have made it through the intake pipeline.
- The mechanism to do this already exists, since we keep track of how many events from a given peer are in the intake pipeline
- In the happy case, where there is no stress on a node, waiting to return the permit won't have any effect
- Since there is 1 permit per peer, and we already wait to perform a sync with a given peer until all events from the previous sync are through the intake pipeline, nothing changes
- Therefore, the change in sync return logic will only come into effect for stressed nodes, which is exactly what we want
- There is a small complication from the orphan buffer, since it is possible for events to remain in the intake pipeline for a while
- Therefore, we would need to start keeping an additional counter (similar to the counter for number of events from a given peer in the intake pipeline), which would represent the number of events from a given peer in the orphan buffer
- The logic for syncing another time with the same peer would remain as currently implemented (don't sync with the same peer again if events from that peer are still in the orphan buffer)
- The logic for returning sync permits during times of stress _would_ consider number of events from the peer in the orphan buffer (i.e., the permit would be released when `events_from_peer_in_intake - events_from_peer_in_orphan_buffer == 0`
Contributor guide
Research direction
Start by locating the sync permit return logic and the existing per-peer intake-pipeline event counter described in the issue. Then trace orphan-buffer handling and related tests; done means stressed nodes retain permits until the relevant peer events have cleared intake and orphan-buffer conditions, without changing the unstressed path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100