hiero-ledger / hiero-ledger/hiero-consensus-node
00287: CATASTROPHIC_FAILURE "DefaultIssDetector: Invalid State Signature (ISS): this node has the wrong hash"
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
Nightly 'main' run failed due to:
https://perf.analytics.eng.hashgraph.io/ephemeral/test_runs/refs/tags/build-00287_SDPT1_312/report/HCSLoadTest/podlog_solo-sdpt-n1/network-node1_logs/swirlds.log
`2026-02-24 11:15:31.366 47495 FATAL EXCEPTION DefaultIssDetector: Invalid State Signature (ISS): this node has the wrong hash for round 161790.`
https://perf.analytics.eng.hashgraph.io/ephemeral/test_runs/refs/tags/build-00287_SDPT1_312/report/HCSLoadTest/podlog_solo-sdpt-n1/network-node1_logs/hgcaa.log
```
2026-02-24 11:15:31.370 WARN 29 FatalIssListenerImpl - ISS detected (type=SELF_ISS, round=161790)
2026-02-24 11:15:31.371 INFO 649 Hedera - HederaNode#0 is CATASTROPHIC_FAILURE
2026-02-24 11:15:31.371 ERROR 669 Hedera - Platform status is now CATASTROPHIC_FAILURE
2026-02-24 11:15:31.371 INFO 247 NettyGrpcServerManager - Shutting down gRPC server on port 50211
2026-02-24 11:15:31.377 INFO 332 NettyGrpcServerManager - gRPC server stopped
2026-02-24 11:15:31.377 INFO 254 NettyGrpcServerManager - Shutting down TLS gRPC server on port 50212
2026-02-24 11:15:31.383 INFO 332 NettyGrpcServerManager - gRPC server stopped
2026-02-24 11:15:31.383 INFO 261 NettyGrpcServerManager - Shutting down node operator gRPC server on port 50213
2026-02-24 11:15:31.383 INFO 332 NettyGrpcServerManager - gRPC server stopped
2026-02-24 11:15:31.383 INFO 672 Hedera - CATASTROPHIC_FAILURE - Shutting down connections to Block Nodes
2026-02-24 11:15:31.387 FATAL 1130 BlockStreamManagerImpl - Awaiting any in-progress round to be closed within PT10S
2026-02-24 11:15:31.834 FATAL 1135 BlockStreamManagerImpl - Block stream fatal shutdown complete
2026-02-24 11:15:34.455 FATAL 334 BlockStreamManagerImpl - Ignoring round 161814 after fatal shutdown request
2026-02-24 11:15:34.455 ERROR 995 BlockStreamManagerImpl - Error occurred while executing task
java.lang.NullPointerException: Cannot invoke "com.hedera.node.app.blocks.BlockItemWriter.writePbjItemAndBytes(com.hedera.hapi.block.stream.BlockItem, com.hedera.pbj.runtime.io.buffer.Bytes)" because "this.this$0.writer" is null
at com.hedera.node.app.blocks.impl.BlockStreamManagerImpl$SequentialTask.onExecute(BlockStreamManagerImpl.java:987)
at org.hiero.base.concurrent.AbstractTask.exec(AbstractTask.java:68)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
```
All logs:
https://perf.analytics.eng.hashgraph.io/ephemeral/test_runs/refs/tags/build-00287_SDPT1_312/report
Contributor guide
Research direction
Start with the nightly run logs and the Invalid State Signature at round 161790, then inspect com.hedera.node.app.blocks.impl.BlockStreamManagerImpl around line 987. Trace the catastrophic-failure shutdown and subsequent writer null pointer; done means identifying and preventing the failure path, with the reported test run no longer producing the fatal error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100