hiero-ledger / hiero-ledger/hiero-consensus-node

00287: CATASTROPHIC_FAILURE "DefaultIssDetector: Invalid State Signature (ISS): this node has the wrong hash"

Open
#23,816 2 comments 0 reactions 0 assignees View on GitHub
Blocking Bug
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

Nightly 'main' run failed due to:
https://perf.analytics.eng.hashgraph.io/ephemeral/test_runs/refs/tags/build-00287_SDPT1_312/report/HCSLoadTest/podlog_solo-sdpt-n1/network-node1_logs/swirlds.log

`2026-02-24 11:15:31.366 47495 FATAL EXCEPTION DefaultIssDetector: Invalid State Signature (ISS): this node has the wrong hash for round 161790.`

https://perf.analytics.eng.hashgraph.io/ephemeral/test_runs/refs/tags/build-00287_SDPT1_312/report/HCSLoadTest/podlog_solo-sdpt-n1/network-node1_logs/hgcaa.log
```
2026-02-24 11:15:31.370 WARN 29 FatalIssListenerImpl - ISS detected (type=SELF_ISS, round=161790)
2026-02-24 11:15:31.371 INFO 649 Hedera - HederaNode#0 is CATASTROPHIC_FAILURE
2026-02-24 11:15:31.371 ERROR 669 Hedera - Platform status is now CATASTROPHIC_FAILURE
2026-02-24 11:15:31.371 INFO 247 NettyGrpcServerManager - Shutting down gRPC server on port 50211
2026-02-24 11:15:31.377 INFO 332 NettyGrpcServerManager - gRPC server stopped
2026-02-24 11:15:31.377 INFO 254 NettyGrpcServerManager - Shutting down TLS gRPC server on port 50212
2026-02-24 11:15:31.383 INFO 332 NettyGrpcServerManager - gRPC server stopped
2026-02-24 11:15:31.383 INFO 261 NettyGrpcServerManager - Shutting down node operator gRPC server on port 50213
2026-02-24 11:15:31.383 INFO 332 NettyGrpcServerManager - gRPC server stopped
2026-02-24 11:15:31.383 INFO 672 Hedera - CATASTROPHIC_FAILURE - Shutting down connections to Block Nodes
2026-02-24 11:15:31.387 FATAL 1130 BlockStreamManagerImpl - Awaiting any in-progress round to be closed within PT10S
2026-02-24 11:15:31.834 FATAL 1135 BlockStreamManagerImpl - Block stream fatal shutdown complete
2026-02-24 11:15:34.455 FATAL 334 BlockStreamManagerImpl - Ignoring round 161814 after fatal shutdown request
2026-02-24 11:15:34.455 ERROR 995 BlockStreamManagerImpl - Error occurred while executing task
java.lang.NullPointerException: Cannot invoke "com.hedera.node.app.blocks.BlockItemWriter.writePbjItemAndBytes(com.hedera.hapi.block.stream.BlockItem, com.hedera.pbj.runtime.io.buffer.Bytes)" because "this.this$0.writer" is null
at com.hedera.node.app.blocks.impl.BlockStreamManagerImpl$SequentialTask.onExecute(BlockStreamManagerImpl.java:987)
at org.hiero.base.concurrent.AbstractTask.exec(AbstractTask.java:68)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
```

All logs:
https://perf.analytics.eng.hashgraph.io/ephemeral/test_runs/refs/tags/build-00287_SDPT1_312/report

Contributor guide

Open the contributing guide

Research direction

Start with the nightly run logs and the Invalid State Signature at round 161790, then inspect com.hedera.node.app.blocks.impl.BlockStreamManagerImpl around line 987. Trace the catastrophic-failure shutdown and subsequent writer null pointer; done means identifying and preventing the failure path, with the reported test run no longer producing the fatal error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.