hiero-ledger / hiero-ledger/hiero-consensus-node

Add retry on state validation fail (faulty download case)

Open
#22,716 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Hedera State Operator
Dominant language
Java
Stars
407
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

Rarely, the state validation can fail with:

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Bad file descriptor
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
	at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:540)
	at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:567)
	at java.base/java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:653)
	at com.hedera.statevalidation.validator.merkledb.ValidateLeafIndexHalfDiskHashMap.validateIndex(ValidateLeafIndexHalfDiskHashMap.java:133)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

(see https://github.com/hashgraph/hedera-state-validator/actions/runs/20437384837/job/58722111156#logs as an example)
If the validation is re-run with the same node and round, it passes, which suggests that sometimes there can be a faulty download, and in this case, the job should be retried.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ValidateLeafIndexHalfDiskHashMap.validateIndex, the method named in the failure trace, and inspect the state-validation job from the linked GitHub Actions run. Reproduce or review validation for the same node and round, then define the retry boundary around the faulty-download failure. Done means a transient validation failure reruns the job and a successful rerun is reported correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ci-cd
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.