hiero-ledger / hiero-ledger/hiero-consensus-node

Quiescence: heartbeat state desync and EPOCH TCT treated as deadline

Open
#25,140 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

**Summary**

While investigating #24398, we tightened the Quiescence HAPI test assertions and the stronger assertions surfaced two latent bugs in the quiescence implementation, plus a design-level interaction with throttling. Fixes are in PR #24625; this issue tracks the underlying problems.

**Investigation**

The original Quiescence test was passing, but only because the assertion it relied on matched a log line emitted at node startup, not a log line proving the node had actually quiesced during the test window. To close that gap we added `assertHgcaaLogContainsTimeframe`, which restricts the log search to the test's own timeframe. With that scoping in place the assertion started failing, and the failures pointed at the production code rather than the test itself.

**Problems**

1. Heartbeat command was not synced back to the caller
Where: `QuiescedHeartbeat`, `QuiescenceController`, `BlockStreamManagerImpl`, `BlockRecordManagerImpl`
When the heartbeat transitioned away from `QUIESCE` (to `DONT_QUIESCE` or `BREAK_QUIESCENCE`), it dispatched the command to the platform but the caller's `lastQuiescenceCommand` stayed at `QUIESCE`. Because the caller only restarts the heartbeat on a transition into `QUIESCE` (`commandNow != lastCommand && commandNow == QUIESCE`), a stale `lastQuiescenceCommand = QUIESCE` silently disabled every subsequent quiescence cycle.
Fix: Centralize dispatch in a new `@Singleton QuiescenceCommands` that owns the `Platform` reference and the last-sent `QuiescenceCommand`.

2. `Instant.EPOCH` TCT was treated as a real deadline
Where: `QuiescenceController`
`tct.minus(config.tctDuration()).isBefore(time.instant())` evaluates to true for an unset/EPOCH TCT, which prematurely forced `DONT_QUIESCE` before any genuine deadline existed.
Fix: short-circuit on `Instant.EPOCH` so an unset TCT does not behave like a missed deadline.

3. Quiescence-induced consensus stall in the steady-state throttle suite
Where: hapiTestStateThrottling suite, specifically ThrottlesContractCallsAsExpected.
With `quiescence.enabled=true`, the test consistently triggers a network-wide consensus stall partway through the run. Subsequent submissions return `PLATFORM_NOT_ACTIVE` and the test ultimately fails with Wrong actual status `UNKNOWN`.
Short-term fix (in #24625): keep `quiescence.enabled=false` for `hapiTestStateThrottling`.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing PR #24625 and the listed classes: QuiescedHeartbeat, QuiescenceController, QuiescenceCommands, BlockStreamManagerImpl, and BlockRecordManagerImpl. Run the tightened Quiescence HAPI assertions and the hapiTestStateThrottling suite, including ThrottlesContractCallsAsExpected; done means heartbeat state remains synchronized, an EPOCH TCT does not act as a deadline, and the throttling test avoids the consensus stall.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.