hiero-ledger / hiero-ledger/hiero-consensus-node
Learner nodes are marked as unhealthy when catching up after reconnect
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
Here is a reconnect scenario:
1. A node falls behind the network and starts a reconnect
2. After reconnect is complete (there may be more than one reconnect steps in a row), the node switches from BEHIND to CHECKING
3. Shortly after that the node is marked as unhealthy and starts a new reconnect cycle
Here is an example: https://preproduction.grafana.hedera-ops.com/d/bdcrx247h03r4d/multi-any-stat-2b-status?from=2024-09-14T16:14:45.229Z&to=2024-09-14T16:34:39.932Z&timezone=utc&var-network=engnet1&var-Datasource=grafanacloud-prom&var-name1=platform_TransactionHandler_busy_fraction&var-name2=platform_TransactionHandler_unhandled_task_count
Check the `platform_TransactionHandler_unhandled_task_count` metric for node 00 (learner). At step 2, the metrics climbs up quickly to 70+, which seems to cause the health monitor to mark the node as unhealthy. This may or may not be correct in general (see discussions in #14422), but seems totally wrong in case of reconnects.
This issue seems to be different from what is described in https://github.com/hashgraph/hedera-services/issues/14422.
Contributor guide
Research direction
Start by tracing the reconnect transition from BEHIND to CHECKING and how the health monitor uses platform_TransactionHandler_unhandled_task_count for learner node 00. Compare the behavior with the discussion in issue #14422 and the linked monitoring example. Done means a learner catching up after reconnect is not incorrectly marked unhealthy, with regression coverage for the scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100