hiero-ledger / hiero-ledger/hiero-consensus-node
Mirror node acceptance tests fail post-restart with "No network stake data found" during mainnet migration test
@mhess-swl is already working on this.
Since Jul 28, 2026.
- Dominant language
- Java
- Stars
- 407
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
Summary
During mainnet migration testing (0.74.3 → 0.75.1, staging-lg environment), mirror node acceptance tests failed post-restart with:
org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: "{"_status":{"messages":[{"detail":"No network stake data found","message":"Not Found"}]}}"
Reran the tests a second time — same error.
Root Cause (suspected)
Migration test froze the network before midnight and restarted it after midnight. A WRB migration took place around that boundary.
Consensus-node observability data (see Diagnostics below) shows migration root hash voting finalized successfully and staking-period reward/fee processing ran without error around the restart, so this does not currently look like a consensus-node halt or crash. It's still unconfirmed whether a NodeStakeUpdate record/block was actually published during the affected run, or whether this is a mirror-node ingestion gap.
Rerunning the migration test with freeze/restart entirely after midnight (i.e., not spanning the midnight boundary) passed without the No network stake data found error.
Environment
- Upgrade path: 0.74.3 → 0.75.1
- Environment: staging-lg
- Failed stage: Post-Restart Validation / Acceptance Tests
Diagnostics
Failing scenario:
ERROR ... o.h.m.t.e.a.s.NetworkFeature Scenario failure:
I verify the network stake <- 404 Not Found: "{"_status":{"messages":[{"detail":"No network stake data found","message":"Not Found"}]}}"
org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: "{"_status":{"messages":[{"detail":"No network stake data found","message":"Not Found"}]}}"
at org.hiero.mirror.test.e2e.acceptance.client.MirrorNodeClient.getNetworkStake(MirrorNodeClient.java:374)
at org.hiero.mirror.test.e2e.acceptance.steps.NetworkFeature.verifyNetworkStake(NetworkFeature.java:37)
at ✽.I verify the network stake(classpath:features/network/network.feature:6)
Test run summary:
Failures (1):
JUnit Platform Suite:AcceptanceTest:Cucumber:Network Coverage Feature:Get network stake
ClasspathResourceSource [classpathResourceName = 'features/network/network.feature', filePosition = FilePosition [line = 5, column = 3]]
=> org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: "{"_status":{"messages":[{"detail":"No network stake data found","message":"Not Found"}]}}"
[ 75 containers found ]
[ 0 containers skipped ]
[ 75 containers started ]
[ 0 containers aborted ]
[ 75 containers successful ]
[ 0 containers failed ]
[ 45 tests found ]
[ 20 tests skipped ]
[ 25 tests started ]
[ 0 tests aborted ]
[ 24 tests successful ]
[ 1 tests failed ]
Consensus-node observability data for the staging-lg hosts during the freeze/restart window (23:40–00:45 UTC) shows no ERROR/WARN entries, migration root hash voting finalizing successfully (>1/3 threshold reached), and staking-period reward/fee processing running normally around the restart.
Full acceptance test log: acceptance-tests-45064.log
For more information, see the internal incident discussion: https://swirldslabs.slack.com/archives/C0BJWK4MQ2X
Reproduction status
- First occurrence: migration test freeze before midnight, restart after midnight — failed with
No network stake data found. - Reran freeze/restart entirely after midnight — passed without the error.
/cc @mhess-swl
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.