hiero-ledger / hiero-ledger/hiero-consensus-node

Child has a missing parent after multiple reconnect

Open
#22,362 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

```
2025-11-28 16:45:18.981 3253 INFO RECONNECT <> ReconnectController: A state was obtained from a peer
2025-11-28 16:45:18.986 3254 INFO RECONNECT <> ReconnectController: The state obtained from a peer was validated
2025-11-28 16:45:19.188 3256 INFO STATE_TO_DISK <> DefaultSavedStateController: Signed state from round 534765 created, will eventually be written to disk, for reason: RECONNECT
2025-11-28 16:45:19.189 3258 INFO PLATFORM_STATUS StatusStateMachine: Platform spent 17.2 m in BEHIND. Now in RECONNECT_COMPLETE {"oldStatus":"BEHIND","newStatus":"RECONNECT_COMPLETE"} [com.swirlds.logging.legacy.payload.PlatformStatusPayload]
2025-11-28 16:45:19.189 3259 INFO STARTUP Shadowgraph: Shadowgraph starting from expiration threshold 534738
2025-11-28 16:45:19.193 3261 INFO STATE_TO_DISK <> SignedStateFileWriter: Started writing round 534765 state to disk. Reason: RECONNECT, directory: /opt/hgcapp/services-hedera/HapiApp2.0/data/saved/com.hedera.services.ServicesMain/27/123/534765
2025-11-28 16:45:19.194 3262 INFO STATE_TO_DISK <> MerkleTreeSnapshotWriter: Creating a snapshot on demand in /opt/hgcapp/services-hedera/HapiApp2.0/data/saved/swirlds-tmp/3 for round 534765
2025-11-28 16:45:19.200 3263 INFO STARTUP PcesFileManager: Due to recent operations on this node, the local preconsensus event stream will have a discontinuity. The last file with the old origin round is 2025-11-28T14+39+36.660229756Z_seq31192_minr528231_maxr528260_orgn491721.pces. All future files will have an origin round of 534765.
2025-11-28 16:45:19.202 3264 INFO RECONNECT <> ReconnectController: Reconnect almost done resuming gossip
2025-11-28 16:45:19.220 3268 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535764,ancientThreshold=535737,expiredThreshold=534763]
2025-11-28 16:45:20.153 3271 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535764,ancientThreshold=535737,expiredThreshold=534763]
2025-11-28 16:45:21.011 3272 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535765,ancientThreshold=535738,expiredThreshold=534764]
2025-11-28 16:45:21.275 3273 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535765,ancientThreshold=535738,expiredThreshold=534764]
2025-11-28 16:45:21.989 3274 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535766,ancientThreshold=535739,expiredThreshold=534765]
2025-11-28 16:45:22.211 3275 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535766,ancientThreshold=535739,expiredThreshold=534765]
2025-11-28 16:45:23.141 3276 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535766,ancientThreshold=535739,expiredThreshold=534765]
2025-11-28 16:45:23.281 3277 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535766,ancientThreshold=535739,expiredThreshold=534765]
2025-11-28 16:45:23.817 3278 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535766,ancientThreshold=535739,expiredThreshold=534765]
2025-11-28 16:45:24.201 3279 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535766,ancientThreshold=535739,expiredThreshold=534765]
2025-11-28 16:45:24.664 3280 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535767,ancientThreshold=535740,expiredThreshold=534766]
2025-11-28 16:45:24.769 3281 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535767,ancientThreshold=535740,expiredThreshold=534766]
2025-11-28 16:45:25.089 3282 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535767,ancientThreshold=535740,expiredThreshold=534766]
2025-11-28 16:45:25.785 3283 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535769,ancientThreshold=535742,expiredThreshold=534768]
2025-11-28 16:45:26.246 3284 INFO RECONNECT <> ReconnectController: Preparing for reconnect, stopping gossip
2025-11-28 16:45:26.246 3285 INFO RECONNECT <> RpcPeerHandler: SELF_FALLEN_BEHIND local ev=EventWindow[latestConsensusRound=534765,ancientThreshold=534738,expiredThreshold=534738] remote ev=EventWindow[latestConsensusRound=535769,ancientThreshold=535742,expiredThreshold=534768]
2025-11-28 16:45:26.246 3286 INFO RECONNECT <> ReconnectController: Preparing for reconnect, start clearing queues
2025-11-28 16:45:26.246 3287 INFO PLATFORM_STATUS StatusStateMachine: Platform spent 7.1 s in RECONNECT_COMPLETE. Now in BEHIND {"oldStatus":"RECONNECT_COMPLETE","newStatus":"BEHIND"} [com.swirlds.logging.legacy.payload.PlatformStatusPayload]
2025-11-28 16:45:26.407 3288 INFO RECONNECT <> ReconnectController: Queues have been cleared
2025-11-28 16:45:26.407 3289 INFO RECONNECT <> ReconnectController: Waiting for a state to be obtained from a peer
2025-11-28 16:45:26.412 3290 ERROR EXCEPTION <> LinkerLogsAndMetrics: Child has a missing parent. This should not be possible. Child: EventDescriptorWrapper[eventDescriptor=EventDescriptor[hash=f1c722ff34bd4be57d0148674f196e90c5111dc80c618a83c5b6bd1431378403bee12928bbd4f1334248a79de1c1ae93, creatorNodeId=4, birthRound=534761], hash=f1c722ff34bd4be57d0148674f196e90c5111dc80c618a83c5b6bd1431378403bee12928bbd4f1334248a79de1c1ae93, creator=4]
sp: EventDescriptorWrapper[eventDescriptor=EventDescriptor[hash=9e4f434e02c69bb0285b7f891b8dacd36c6a213e91fc2a8a8f5d2be5babf88e8385be54531abfa1269b739bf47c4fdda, creatorNodeId=4, birthRound=534761], hash=9e4f434e02c69bb0285b7f891b8dacd36c6a213e91fc2a8a8f5d2be5babf88e8385be54531abfa1269b739bf47c4fdda, creator=4]
op0: EventDescriptorWrapper[eventDescriptor=EventDescriptor[hash=ca8f070e3a8d5996149d7391d7a5e1d1a5edc3494abc3811a4d73db40ed570c5288f9158525dee182fd4d6eaae33a5c4, creatorNodeId=26, birthRound=534761], hash=ca8f070e3a8d5996149d7391d7a5e1d1a5edc3494abc3811a4d73db40ed570c5288f9158525dee182fd4d6eaae33a5c4, creator=26], Parent EventDescriptor: EventDescriptorWrapper[eventDescriptor=EventDescriptor[hash=9e4f434e02c69bb0285b7f891b8dacd36c6a213e91fc2a8a8f5d2be5babf88e8385be54531abfa1269b739bf47c4fdda, creatorNodeId=4, birthRound=534761], hash=9e4f434e02c69bb0285b7f891b8dacd36c6a213e91fc2a8a8f5d2be5babf88e8385be54531abfa1269b739bf47c4fdda, creator=4]
2025-11-28 16:45:26.417 3291 INFO RECONNECT <> ReconnectStatePeerProtocol: Starting reconnect in role of the receiver. {"receiving":true,"nodeId":27,"otherNodeId":25,"round":534765} [com.swirlds.logging.legacy.payload.ReconnectStartPayload]
2025-11-28 16:45:26.418 3292 INFO RECONNECT <> ReconnectStateLearner: Receiving signed state signatures
2025-11-28 16:45:26.427 3298 ERROR EXCEPTION DefaultUncaughtExceptionHandler: Uncaught exception in gossip
java.lang.IllegalStateException: Initial event window not set
at com.swirlds.platform.gossip.shadowgraph.Shadowgraph.addEvent(Shadowgraph.java:506)
at com.swirlds.platform.gossip.shadowgraph.AbstractShadowgraphSynchronizer.addEvent(AbstractShadowgraphSynchronizer.java:253)
at com.swirlds.component.framework.wires.input.BindableInputWire.lambda$bindConsumer$0(BindableInputWire.java:77)
at com.swirlds.component.framework.schedulers.internal.SequentialTask.onExecute(SequentialTask.java:77)
at org.hiero.base.concurrent.AbstractTask.exec(AbstractTask.java:68)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
2025-11-28 16:45:26.427 3296 ERROR EXCEPTION DefaultUncaughtExceptionHandler: Uncaught exception in gossip
```

Contributor guide

Open the contributing guide

Research direction

Start with the stack trace at Shadowgraph.java:506 and follow addEvent() through AbstractShadowgraphSynchronizer.java:253. Review the reconnectController sequence around the missing-parent error and the subsequent “Initial event window not set” exception. Done means repeated reconnects no longer produce either failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.