hiero-ledger / hiero-ledger/hiero-consensus-node

Mirror node acceptance tests fail post-restart with "No network stake data found" during mainnet migration test

Open
#26,595 0 comments 0 reactions 1 assignee View on GitHub

@mhess-swl is already working on this.

Since Jul 28, 2026.

Execution Team
Dominant language
Java
Stars
407
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

Summary

During mainnet migration testing (0.74.3 → 0.75.1, staging-lg environment), mirror node acceptance tests failed post-restart with:

org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: "{"_status":{"messages":[{"detail":"No network stake data found","message":"Not Found"}]}}"

Reran the tests a second time — same error.

Root Cause (suspected)

Migration test froze the network before midnight and restarted it after midnight. A WRB migration took place around that boundary.

Consensus-node observability data (see Diagnostics below) shows migration root hash voting finalized successfully and staking-period reward/fee processing ran without error around the restart, so this does not currently look like a consensus-node halt or crash. It's still unconfirmed whether a NodeStakeUpdate record/block was actually published during the affected run, or whether this is a mirror-node ingestion gap.

Rerunning the migration test with freeze/restart entirely after midnight (i.e., not spanning the midnight boundary) passed without the No network stake data found error.

Environment

  • Upgrade path: 0.74.3 → 0.75.1
  • Environment: staging-lg
  • Failed stage: Post-Restart Validation / Acceptance Tests

Diagnostics

Failing scenario:

ERROR ... o.h.m.t.e.a.s.NetworkFeature Scenario failure:
I verify the network stake <- 404 Not Found: "{"_status":{"messages":[{"detail":"No network stake data found","message":"Not Found"}]}}"
 org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: "{"_status":{"messages":[{"detail":"No network stake data found","message":"Not Found"}]}}"
	at org.hiero.mirror.test.e2e.acceptance.client.MirrorNodeClient.getNetworkStake(MirrorNodeClient.java:374)
	at org.hiero.mirror.test.e2e.acceptance.steps.NetworkFeature.verifyNetworkStake(NetworkFeature.java:37)
	at ✽.I verify the network stake(classpath:features/network/network.feature:6)

Test run summary:

Failures (1):
  JUnit Platform Suite:AcceptanceTest:Cucumber:Network Coverage Feature:Get network stake
    ClasspathResourceSource [classpathResourceName = 'features/network/network.feature', filePosition = FilePosition [line = 5, column = 3]]
    => org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: "{"_status":{"messages":[{"detail":"No network stake data found","message":"Not Found"}]}}"

[        75 containers found      ]
[         0 containers skipped    ]
[        75 containers started    ]
[         0 containers aborted    ]
[        75 containers successful ]
[         0 containers failed     ]
[        45 tests found           ]
[        20 tests skipped         ]
[        25 tests started         ]
[         0 tests aborted         ]
[        24 tests successful      ]
[         1 tests failed          ]

Consensus-node observability data for the staging-lg hosts during the freeze/restart window (23:40–00:45 UTC) shows no ERROR/WARN entries, migration root hash voting finalizing successfully (>1/3 threshold reached), and staking-period reward/fee processing running normally around the restart.

Full acceptance test log: acceptance-tests-45064.log

For more information, see the internal incident discussion: https://swirldslabs.slack.com/archives/C0BJWK4MQ2X

Reproduction status

  • First occurrence: migration test freeze before midnight, restart after midnight — failed with No network stake data found.
  • Reran freeze/restart entirely after midnight — passed without the error.

/cc @mhess-swl

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.