hiero-ledger / hiero-ledger/hiero-consensus-node
bug: consensus node re-keys TSS after restore from freeze state, breaking block-node verification
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
## Background
When a Hiero consensus node (CN) is restored from a freeze-based saved state (as done by the `solo` multicluster backup-restore flow), CN performs a TSS re-keying cycle shortly after startup. This re-keying changes the TSS keys used to sign blocks, invalidating any block-node `tss-bootstrap-roster.json` that was captured before the restore.
**Observed behaviour (CN v0.74, solo multicluster-backup-restore example):**
1. Initial deploy: CN establishes TSS keys around block ~123. Block-node captures `tss-bootstrap-roster.json` in its `application-state/`.
2. Backup is taken (block-node tip ≈ block 148).
3. Cluster is destroyed and recreated. Block-node archive is restored (blocks 0–148). `tss-bootstrap-roster.json` from the backup is seeded into block-node's `application-state/`.
4. CN starts from the restored freeze state and streams blocks to block-node.
5. Blocks 149–153 pass TSS verification (old keys still match).
6. **Block 154: TSS re-keying completes. Block 154's proof uses new TSS keys. Block-node fails verification with `BAD_BLOCK_PROOF`.** Block-node rolls back to block 148 and enters an unrecoverable `RESEND(154)` loop.
Block-node log at failure:
```
WARNING Block 154 state proof verification failed with success=false
FINE [N0-STR13] Handler 12 handling failed verification for block 154
FINE [N0-STR13] Handler 12 ending with code BAD_BLOCK_PROOF
```
## Root Cause (from source analysis)
The trigger is **`StakePeriodChanges.processSideEffects()`** in `hedera-node/hedera-app/src/main/java/com/hedera/node/app/workflows/handle/steps/StakePeriodChanges.java` (lines ~157–180). At every staking period boundary (midnight UTC), it unconditionally stores a new weight-rotation candidate roster:
```java
if (rosterStore.getCandidateRosterHash() == null || rosterStore.candidateIsWeightRotation()) {
rosterStore.putCandidateRoster(reweightedRoster); // fires at period boundary
}
```
`putCandidateRoster()` has no hash-equality guard — it stores the roster even when staking weights haven't changed, because minor weight drift can produce a different hash. The moment `candidateRosterHash != null`, `reconcileTssState()` (called every round from `HandleWorkflow.java` line ~366) evaluates `ActiveRosters.from()` into **TRANSITION phase**, which causes `WritableHintsStoreImpl.getOrCreateConstruction()` to create a new hinTS construction → re-keying starts.
**Timeline match:** After restore, CN produces blocks normally with the old TSS keys. Re-keying only starts when the **first staking period boundary** (midnight UTC) is crossed after restart — explaining the "few blocks" delay before block 154 (which landed ~3–4 minutes into restore in CI runs).
**Startup schema does NOT force reset:** `V073HintsSchema.restart()` restores (not resets) the existing signing context from saved state. The re-keying is not triggered at startup itself — only at the next staking boundary.
## Key Files
| File | Method | Role |
|---|---|---|
| `hedera-node/hedera-app/…/handle/steps/StakePeriodChanges.java` | `processSideEffects()` | Sets weight-rotation candidate roster at period boundary — the trigger |
| `hedera-node/hedera-roster-service-impl/…/ActiveRosters.java` | `from()` | Determines phase; TRANSITION fires when `candidateRosterHash` is non-null |
| `hedera-node/hedera-app/…/hints/impl/WritableHintsStoreImpl.java` | `getOrCreateConstruction()` / `updateForNewConstruction()` | Creates the new hinTS construction that starts re-keying |
| `hedera-node/hedera-app/…/hints/impl/HintsServiceImpl.java` | `reconcile()` | Called every round; drives construction creation and advancement |
| `hedera-node/hedera-app/…/hints/schemas/V073HintsSchema.java` | `restart()` | Restores (does not reset) the active signing context from saved state |
## Proposed Fix
Add a hash-equality guard in `StakePeriodChanges.processSideEffects()` before calling `putCandidateRoster()`: skip the call when the reweighted roster hash equals the current active roster hash, preventing unnecessary re-keying when weights are unchanged.
Alternatively, if re-keying on every weight rotation is intentional by design, block-node should refresh its `tss-bootstrap-roster.json` when a TSS handoff completes — this could be driven via the `onFinishedConstruction` callback registered in `HandleWorkflow.configureTssCallbacks()` (~line 1070).
## Impact
This makes it impossible for block-node to use its backed-up `tss-bootstrap-roster.json` after restoring a CN cluster from freeze state. Block-node cannot verify post-restore blocks, causing a permanent stream failure.
**Current workaround in solo (hiero-ledger/solo#4985):** Do not seed `tss-bootstrap-roster.json` on restore. Block-node then runs without TSS verification (matching fresh initial-deploy behaviour). This is a temporary workaround — long-term, block-node should be able to verify blocks correctly after restore.
## Acceptance Criteria
1. CN restored from a freeze-based saved state does NOT perform a TSS re-keying cycle unless staking weights actually changed (roster hash differs from the active roster).
2. If re-keying is necessary, CN notifies block-node of the new TSS state before the first re-keyed block is delivered (via `onFinishedConstruction` or equivalent).
3. The solo multicluster backup-restore example can seed `tss-bootstrap-roster.json` from backup and have block-node verify all post-restore blocks without `BAD_BLOCK_PROOF` errors.
## Dependencies
- hiero-ledger/solo#4985 (workaround PR)
- hiero-ledger/hiero-consensus-node#25389 (related TSS/CHECKING bug)
## Environment
- CN version: v0.74
- Block-node version: 0.37.1
- Solo version: main branch (hiero-ledger/solo#4985)
- Reproduced in: GitHub Actions CI (solo multicluster-backup-restore example test)
Contributor guide
Research direction
Start with StakePeriodChanges.processSideEffects() and trace candidate-roster handling through ActiveRosters.from(), WritableHintsStoreImpl, and HintsServiceImpl.reconcile(). Read V073HintsSchema.restart() to confirm restored signing context, then reproduce the solo multicluster backup-restore example. Done means restored block-nodes verify post-restore blocks without BAD_BLOCK_PROOF, and re-keying occurs only when the roster hash changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- cryptography, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100