hiero-ledger / hiero-ledger/hiero-consensus-node

bug: consensus node re-keys TSS after restore from freeze state, breaking block-node verification

Open
#26,299 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

## Background

When a Hiero consensus node (CN) is restored from a freeze-based saved state (as done by the `solo` multicluster backup-restore flow), CN performs a TSS re-keying cycle shortly after startup. This re-keying changes the TSS keys used to sign blocks, invalidating any block-node `tss-bootstrap-roster.json` that was captured before the restore.

**Observed behaviour (CN v0.74, solo multicluster-backup-restore example):**

1. Initial deploy: CN establishes TSS keys around block ~123. Block-node captures `tss-bootstrap-roster.json` in its `application-state/`.
2. Backup is taken (block-node tip ≈ block 148).
3. Cluster is destroyed and recreated. Block-node archive is restored (blocks 0–148). `tss-bootstrap-roster.json` from the backup is seeded into block-node's `application-state/`.
4. CN starts from the restored freeze state and streams blocks to block-node.
5. Blocks 149–153 pass TSS verification (old keys still match).
6. **Block 154: TSS re-keying completes. Block 154's proof uses new TSS keys. Block-node fails verification with `BAD_BLOCK_PROOF`.** Block-node rolls back to block 148 and enters an unrecoverable `RESEND(154)` loop.

Block-node log at failure:
```
WARNING Block 154 state proof verification failed with success=false
FINE [N0-STR13] Handler 12 handling failed verification for block 154
FINE [N0-STR13] Handler 12 ending with code BAD_BLOCK_PROOF
```

## Root Cause (from source analysis)

The trigger is **`StakePeriodChanges.processSideEffects()`** in `hedera-node/hedera-app/src/main/java/com/hedera/node/app/workflows/handle/steps/StakePeriodChanges.java` (lines ~157–180). At every staking period boundary (midnight UTC), it unconditionally stores a new weight-rotation candidate roster:

```java
if (rosterStore.getCandidateRosterHash() == null || rosterStore.candidateIsWeightRotation()) {
rosterStore.putCandidateRoster(reweightedRoster); // fires at period boundary
}
```

`putCandidateRoster()` has no hash-equality guard — it stores the roster even when staking weights haven't changed, because minor weight drift can produce a different hash. The moment `candidateRosterHash != null`, `reconcileTssState()` (called every round from `HandleWorkflow.java` line ~366) evaluates `ActiveRosters.from()` into **TRANSITION phase**, which causes `WritableHintsStoreImpl.getOrCreateConstruction()` to create a new hinTS construction → re-keying starts.

**Timeline match:** After restore, CN produces blocks normally with the old TSS keys. Re-keying only starts when the **first staking period boundary** (midnight UTC) is crossed after restart — explaining the "few blocks" delay before block 154 (which landed ~3–4 minutes into restore in CI runs).

**Startup schema does NOT force reset:** `V073HintsSchema.restart()` restores (not resets) the existing signing context from saved state. The re-keying is not triggered at startup itself — only at the next staking boundary.

## Key Files

| File | Method | Role |
|---|---|---|
| `hedera-node/hedera-app/…/handle/steps/StakePeriodChanges.java` | `processSideEffects()` | Sets weight-rotation candidate roster at period boundary — the trigger |
| `hedera-node/hedera-roster-service-impl/…/ActiveRosters.java` | `from()` | Determines phase; TRANSITION fires when `candidateRosterHash` is non-null |
| `hedera-node/hedera-app/…/hints/impl/WritableHintsStoreImpl.java` | `getOrCreateConstruction()` / `updateForNewConstruction()` | Creates the new hinTS construction that starts re-keying |
| `hedera-node/hedera-app/…/hints/impl/HintsServiceImpl.java` | `reconcile()` | Called every round; drives construction creation and advancement |
| `hedera-node/hedera-app/…/hints/schemas/V073HintsSchema.java` | `restart()` | Restores (does not reset) the active signing context from saved state |

## Proposed Fix

Add a hash-equality guard in `StakePeriodChanges.processSideEffects()` before calling `putCandidateRoster()`: skip the call when the reweighted roster hash equals the current active roster hash, preventing unnecessary re-keying when weights are unchanged.

Alternatively, if re-keying on every weight rotation is intentional by design, block-node should refresh its `tss-bootstrap-roster.json` when a TSS handoff completes — this could be driven via the `onFinishedConstruction` callback registered in `HandleWorkflow.configureTssCallbacks()` (~line 1070).

## Impact

This makes it impossible for block-node to use its backed-up `tss-bootstrap-roster.json` after restoring a CN cluster from freeze state. Block-node cannot verify post-restore blocks, causing a permanent stream failure.

**Current workaround in solo (hiero-ledger/solo#4985):** Do not seed `tss-bootstrap-roster.json` on restore. Block-node then runs without TSS verification (matching fresh initial-deploy behaviour). This is a temporary workaround — long-term, block-node should be able to verify blocks correctly after restore.

## Acceptance Criteria

1. CN restored from a freeze-based saved state does NOT perform a TSS re-keying cycle unless staking weights actually changed (roster hash differs from the active roster).
2. If re-keying is necessary, CN notifies block-node of the new TSS state before the first re-keyed block is delivered (via `onFinishedConstruction` or equivalent).
3. The solo multicluster backup-restore example can seed `tss-bootstrap-roster.json` from backup and have block-node verify all post-restore blocks without `BAD_BLOCK_PROOF` errors.

## Dependencies

- hiero-ledger/solo#4985 (workaround PR)
- hiero-ledger/hiero-consensus-node#25389 (related TSS/CHECKING bug)

## Environment

- CN version: v0.74
- Block-node version: 0.37.1
- Solo version: main branch (hiero-ledger/solo#4985)
- Reproduced in: GitHub Actions CI (solo multicluster-backup-restore example test)

Contributor guide

Open the contributing guide

Research direction

Start with StakePeriodChanges.processSideEffects() and trace candidate-roster handling through ActiveRosters.from(), WritableHintsStoreImpl, and HintsServiceImpl.reconcile(). Read V073HintsSchema.restart() to confirm restored signing context, then reproduce the solo multicluster backup-restore example. Done means restored block-nodes verify post-restore blocks without BAD_BLOCK_PROOF, and re-keying occurs only when the roster hash changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cryptography, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.