hiero-ledger / hiero-ledger/hiero-consensus-node
Bug: Node stuck in WAITING_FOR_LEDGER_ID after restart with historyEnabled=true
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
# Bug: Node stuck in WAITING_FOR_LEDGER_ID after restart with historyEnabled=true
## Summary
After a freeze/restart with `tss.historyEnabled=true`, the node permanently returns `WAITING_FOR_LEDGER_ID` for all gRPC transactions. The signer never becomes ready again. Block signing also stops (`shouldCloseBlock()` returns false). The node appears healthy (ACTIVE status, ledger ID visible in state) but is unable to process any work.
This occurs regardless of `tss.wrapsEnabled`.
## Reproduction
Run `hapiTestRestart` with these properties:
```
tss.hintsEnabled=true
tss.historyEnabled=true
tss.forceMockSignatures=false
```
After the freeze/restart cycle, `StreamValidationTest.streamsAreValid` fails:
```
Wrong precheck status for FileUpdate — Expected OK, actual WAITING_FOR_LEDGER_ID
```
Other post-restart tests (`QuiesceThenMixedOpsRestartTest`, `JumpstartFileSuite`) fail with the same error.
## Observed Behavior
**Before restart (works correctly):**
1. Schnorr genesis construction # 1 completes
2. `TssBlockHashSigner` logs "TSS protocol ready to sign blocks"
3. `HandleWorkflow` logs "Set ledger id to '...'"
4. Blocks are produced and signed normally
**After restart:**
5. Nodes restart, reach ACTIVE status
6. Pending blocks are recovered from disk (e.g., "Recovered pending block # 49")
7. No "TSS protocol ready to sign blocks" message appears
8. All gRPC transactions return `WAITING_FOR_LEDGER_ID`
9. No new blocks are produced
## Analysis
`TssBlockHashSigner.isReady()` returns false because `historyService.isReady()` returns false. `HistoryServiceImpl.isReady()` checks an in-memory field `historyProof` which is null after restart.
`V071HistorySchema.restart()` (line 108) is designed to restore this field from the `ACTIVE_PROOF_CONSTRUCTION` state singleton. However, it does not succeed. Two failure paths were identified:
## Impact
Any network with `tss.historyEnabled=true` cannot survive a restart. After restart:
- All gRPC transactions fail with `WAITING_FOR_LEDGER_ID`
- Block production stops
- The node is functionally dead despite showing ACTIVE status
Contributor guide
Research direction
Start at V071HistorySchema.restart() around line 108 and inspect how the ACTIVE_PROOF_CONSTRUCTION state should restore historyProof. Trace HistoryServiceImpl.isReady() and TssBlockHashSigner.isReady(), then reproduce with hapiTestRestart using the listed properties and failing restart tests. Done means the node becomes ready after restart, transactions no longer return WAITING_FOR_LEDGER_ID, and block production resumes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100