hiero-ledger / hiero-ledger/hiero-consensus-node

Learner nodes are marked as unhealthy when catching up after reconnect

Open
#15,565 1 comment 0 reactions 0 assignees View on GitHub
Performance Platform Platform Reconnect
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

Here is a reconnect scenario:

1. A node falls behind the network and starts a reconnect
2. After reconnect is complete (there may be more than one reconnect steps in a row), the node switches from BEHIND to CHECKING
3. Shortly after that the node is marked as unhealthy and starts a new reconnect cycle

Here is an example: https://preproduction.grafana.hedera-ops.com/d/bdcrx247h03r4d/multi-any-stat-2b-status?from=2024-09-14T16:14:45.229Z&to=2024-09-14T16:34:39.932Z&timezone=utc&var-network=engnet1&var-Datasource=grafanacloud-prom&var-name1=platform_TransactionHandler_busy_fraction&var-name2=platform_TransactionHandler_unhandled_task_count

Check the `platform_TransactionHandler_unhandled_task_count` metric for node 00 (learner). At step 2, the metrics climbs up quickly to 70+, which seems to cause the health monitor to mark the node as unhealthy. This may or may not be correct in general (see discussions in #14422), but seems totally wrong in case of reconnects.

This issue seems to be different from what is described in https://github.com/hashgraph/hedera-services/issues/14422.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the reconnect transition from BEHIND to CHECKING and how the health monitor uses platform_TransactionHandler_unhandled_task_count for learner node 00. Compare the behavior with the discussion in issue #14422 and the linked monitoring example. Done means a learner catching up after reconnect is not incorrectly marked unhealthy, with regression coverage for the scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.