hiero-ledger / hiero-ledger/hiero-consensus-node

Record stream hash continuity can break across freeze/restart (saved state hash diverges from record-file chain anchor)

Open
#25,486 0 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

## Summary
During a freeze/restart boundary, CN can restart from a saved-state running hash that does not match the last hash anchor accepted by mirror importer from record files.

Result: post-restart `.rcd.gz` files are internally valid and signed by both nodes, but mirror importer fails with running-hash mismatch for every new file.

## Expected behavior
After freeze + restart from saved state, the first post-restart record file should chain from the last pre-freeze imported file hash (or there should be an explicit/compatible boundary marker), so importer can continue.

## Actual behavior
Importer remains anchored at pre-freeze file `2026-05-14T14_16_51.212534000Z.rcd.gz` and rejects all later files:

- `Expected = 1e4afae8323276e7f53cc093ead6b9de2cbdab520e419047be4abb6e961fc479862b5b00b9c71f8cb3cfb860aa4adafc`
- first failing file in batch: `2026-05-14T14_18_39.095521000Z.rcd.gz`
- repeated errors: `None of the data files could be verified`

This occurs for signatures from both node0 and node1 (consensus reached).

## Repro context
- Local kind cluster (`kind-solo-e2e`), 2 CN nodes + mirror importer
- Freeze command, restart from saved state, then resume traffic
- Date observed: 2026-05-14

## Evidence
### CN logs (both nodes)
- Freeze round reached and stream writer stopped:
- `Submitting freeze period entered action for consensus round: 3250`
- `TimestampStreamFileWriter finished writing the last object, is stopped`
- Restart loads saved state at freeze round:
- `Loading signed state from disk .../saved/.../3250`
- `Platform has loaded a saved state {"round":3250,"consensusTimestamp":"2026-05-14T14:19:53.738201Z"}`

### MinIO object presence
`record0.0.3` contains all boundary files:
- `2026-05-14T14_16_51.212534000Z.rcd.gz`
- `2026-05-14T14_18_39.095521000Z.rcd.gz`
- `2026-05-14T14_19_13.179527000Z.rcd.gz`
(and corresponding `.rcd_sig`)

### Mirror importer errors
Repeated for both node signatures:
- `Running hash mismatch for file 2026-05-14T14_18_39.095521000Z.rcd.gz. Expected = 1e4afae8..., Actual = b05974f4...`
- `None of the data files could be verified`
- Same pattern continues for all subsequent files (`14_19_*`).

## Code paths that appear to allow this divergence
### CN
1. Restart initializes stream producer from `RUNNING_HASHES_STATE_ID`:
- `BlockRecordManagerImpl` constructor reads running hash state and calls `streamFileProducer.initRunningHash(lastRunningHashes)`.

2. Per-round state update commits running hash:
- `BlockRecordManagerImpl.endRound()` writes `RUNNING_HASHES_STATE_ID` from `streamFileProducer.getRunningHash()`.

3. File closure is conditional by consensus time window:
- `BlockRecordManagerImpl.closeCurrentRecordFileIfConsTimeElapsed()` only closes when period elapsed.

4. Concurrent producer can advance running hash even without active writer:
- `StreamFileProducerConcurrent.writeRecordStreamItems()` updates `lastRecordHashingResult` before checking `currentRecordFileWriter == null`, and then may return while "ignoring serialized items" for writer path.

### Mirror
- Importer correctly enforces chain continuity by comparing next file `previousHash` to last imported hash:
- `Downloader.verify()` / `verifyHashChain()`.

## Suspected invariant gap
At freeze/restart boundary, `saved-state runningHash` and `last closed/uploaded record-file end hash` can diverge.

If true, this is a CN-side continuity bug at boundary semantics for legacy record stream + importer.

## Request
Please confirm whether this invariant is expected to hold:
- `state RUNNING_HASHES runningHash == hash anchor of next post-restart record file expected by importer`

If yes, this appears to be a bug and likely needs a boundary-safe flush/close + state alignment guarantee at freeze/restart.

Contributor guide

Open the contributing guide

Research direction

Start with BlockRecordManagerImpl, including its constructor, endRound(), and closeCurrentRecordFileIfConsTimeElapsed(), then inspect StreamFileProducerConcurrent.writeRecordStreamItems() and the mirror Downloader.verify()/verifyHashChain() paths. Reproduce the freeze, saved-state restart, and importer failure in the kind-solo-e2e context, comparing the saved running hash with the last imported file anchor. Done means the post-restart chain continues without importer hash mismatches or has an explicitly compatible boundary marker.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.