hiero-ledger / hiero-ledger/hiero-consensus-node

Platform exposes no signal for "round N is fully signed and safe to export" — callers must poll and hash the filesystem

Open
#26,950 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

## Background

Solo (`hiero-ledger/solo`) needs to safely export a consensus node's saved state from disk for `consensus state download` / `config ops backup` — used both for bootstrapping a new node from an existing node's state and for disaster-recovery backup/restore workflows (see [hiero-ledger/solo#5375](https://github.com/hiero-ledger/solo/pull/5375)). Before zipping up `data/saved`, the caller needs to know that the round it's about to export has finished collecting all validator signatures and nothing is still being written to disk for that round.

There is currently no platform-exposed signal for this. The two status values that come closest don't actually cover it:

- `FREEZE_COMPLETE` only applies when the network was explicitly frozen, and says nothing about whether a *periodic* signed snapshot (taken while the network stays live) has finished being written.
- For a plain `STOPPED` (non-frozen) deployment there's no target status at all — the caller just wants "the newest round that finished signing," which isn't something any `PlatformStatus` value identifies.

As a workaround, Solo polls the container's filesystem in a loop ([`resources/wait-for-stable-saved-state.sh`](https://github.com/hiero-ledger/solo/blob/02379-D-start-saved-state/resources/wait-for-stable-saved-state.sh) + `NetworkNodes.waitForStableSavedState`, see [hiero-ledger/solo#5375](https://github.com/hiero-ledger/solo/pull/5375)):

1. Reads each round directory's `stateMetadata.txt` and compares `SIGNING_WEIGHT_SUM` to `TOTAL_WEIGHT` to guess whether that round's signatures are complete.
2. Fingerprints the entire `data/saved` tree (originally by hashing every file's contents; changed to size+mtime after review feedback, since hashing full content on every 2s poll for up to 180 polls adds real I/O load exactly while the node is trying to quiesce) and waits for the fingerprint to stay stable across 3 consecutive polls before treating a round as "done."
3. Falls back to the newest fully signed *non-freeze* round if a requested freeze round never becomes fully signed within the poll budget (~6 minutes) — which, in practice, was sometimes the *only* outcome, see below.

Separately, in the same PR we observed a round created by a `FREEZE_ONLY` transaction remain permanently un-signed (`SIGNING_WEIGHT_SUM: 0`, `TOTAL_WEIGHT: 1000`) even after waiting several extra minutes past the freeze, on a small (2-node) test network. This looks adjacent to #11421 ("Periodic snapshots during PCES replay don't get signatures"), but that issue is specifically about snapshots chosen *during PCES replay*; what we saw was a live, non-replay freeze round on an otherwise normally-running network, so it may be a distinct manifestation worth confirming rather than assuming it's the same root cause.

## Proposed Solution

1. Expose an explicit signal (a `PlatformStatus`-adjacent value, a metrics-port gauge, or a per-round "signing complete" event/callback) that answers "round N has collected all its signatures and its on-disk saved-state directory is finished being written — it is safe to read/export now," without requiring the caller to poll and hash the filesystem.
2. The signal should cover both a frozen deployment (the freeze round specifically) and a live/stopped deployment (the newest signed round), since both are legitimate export scenarios Solo relies on.
3. Document the expected latency between a round reaching consensus and its signatures being considered "complete," so callers can size reasonable timeouts instead of guessing (Solo currently polls for up to ~6 minutes per node).

## Acceptance Criteria

1. An external caller can determine, without filesystem polling/hashing, that a specific round's saved state is fully signed and stable on disk.
2. The mechanism works for both a frozen network (freeze round) and a stopped-but-not-frozen network (latest signed round).
3. Expected signing latency (or an explicit "will never sign" condition) is documented or discoverable.

## Dependencies

1. [hiero-ledger/solo#5375](https://github.com/hiero-ledger/solo/pull/5375) — the PR that hit this gap and added the filesystem-polling workaround (see review discussion on `src/core/network-nodes.ts` and `resources/wait-for-stable-saved-state.sh`).

## Related / Adjacent Issues

- #11421 — "Periodic snapshots during PCES replay don't get signatures" (related but distinct: that's about replay-time snapshots never signing; this issue is about live rounds with no way to know signing status short of polling).
- #21606, #26535 — saved-state/`stateMetadata.txt` round integrity issues; adjacent but about folder/round-number consistency rather than signal exposure.

## Environment

- Consensus node version: v0.75.1
- Observed on a 2-node Solo-managed local kind cluster and in Solo's GitHub Actions CI (`hiero-ledger/solo`)

Contributor guide

Open the contributing guide

Research direction

Start with resources/wait-for-stable-saved-state.sh and NetworkNodes.waitForStableSavedState to understand the current filesystem-polling workaround, then trace the PlatformStatus-adjacent signing boundary described in the issue. Done means an external caller can identify a fully signed, stable round for both frozen and stopped deployments, with signing latency or a never-signs condition documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, shell
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.