Single node stuck in CHECKING indefinitely when peers are ACTIVE — gossip isolation deadlock

Summary

A single consensus node that enters CHECKING while all other peers remain ACTIVE becomes permanently stuck — zero gossip events received, zero syncs, zero rounds advancing. The node's gossip layer is completely dead despite confirmed TCP connectivity to all peers. The only recovery is restarting all nodes simultaneously so they enter CHECKING together.

This was reproduced deterministically on a 5-node private network running Hedera v0.68.6 (consensus node image ghcr.io/hashgraph/solo-containers/ubi8-init-java21:0.39.1).

Related Issues

This appears to be in the same class of CHECKING/reconnect instability as:

• #23322 — node4 goes in CHECKING (open, 2026-02-04)
• #8505 — node stuck in CHECKING for >20 minutes, could not reconnect
• #9317 — status swings CHECKING/ACTIVE after restart
• #17526 — frequent ACTIVE→CHECKING oscillation with PLATFORM_NOT_ACTIVE symptoms
• #18656 / PR #18657 — reconnect scenario where nodes expected ACTIVE fall into CHECKING
• #22553 — reconnect teacher backpressure causing ACTIVE/CHECKING switching

However, the distinguishing characteristic of this report is that the node is permanently stuck with zero gossip activity — not oscillating between states.

Environment

• Platform version: v0.68.6
• Network: 5-node private testnet (Solo v0.54.0 deployment)
• Infrastructure: All 5 nodes on a single bare-metal host (K3s / Kubernetes), 64-core, 755GB RAM
• JVM: -Xms64g -Xmx64g, G1GC with G1HeapRegionSize=16m, InitiatingHeapOccupancyPercent=35
• CPU per node: request=8, limit=16

Reproduction Steps

1. Start a 5-node network. Wait until all 5 nodes are ACTIVE and producing blocks.
2. Delete a single consensus node pod (e.g., node 5) while the other 4 remain running.
3. Node 5 restarts, loads saved state, replays PCES, transitions STARTING_UP → REPLAYING_EVENTS → OBSERVING → CHECKING.
4. Node 5 remains stuck in CHECKING indefinitely.

This was triggered by a staggered restart timer (one node restarted per day for heap age desynchronization). The restarted node (node 5) entered CHECKING at 05:30:42 UTC and never recovered. It was still stuck 5+ hours later when manual investigation began.

Observed Behavior

Platform status timeline (node 5)

05:30:31.570  STARTING_UP → REPLAYING_EVENTS (88ms)
05:30:32.220  REPLAYING_EVENTS → OBSERVING (650ms, replayed 2792 PCES events)
05:30:42.281  OBSERVING → CHECKING (10.1s in OBSERVING)
              [NO FURTHER STATUS CHANGES — stuck for 5+ hours]

Gossip metrics comparison (live snapshot during stuck period)

Metric	Node 5 (stuck CHECKING)	Node 1 (healthy ACTIVE)
eventRecRate	1.000	341.008
acceptedSyncRequests_per_sec	0.00	62.94
incomingSyncRequests_per_sec	0.000	0.222
rounds_per_sec	0	3
sync_per_secR	0.000	0.014
tips_per_sync	0	918,434
syncs_in_progress	0.00	565.00

Node 5 shows complete gossip isolation — not a single event received, not a single sync initiated or accepted.

Connectivity

• DNS resolution: All 5 service names resolve correctly.
• TCP: Gossip port 50111 reachable on all 4 peer nodes from node 5.
• No TLS/SSL/handshake/certificate errors in any log.

Log behavior

• swirlds.log stopped writing entirely after the CHECKING entry (line count remained static for hours).
• hgcaa.log continued growing (BlockNodeConnectionManager retries), proving the JVM process is alive.
• ReconnectController logged "Starting the ReconnectController" but never initiated a reconnect.

Recovery Attempts (All Failed)

Three different approaches were tried for node 5 alone — all produced identical behavior:

Attempt	State Configuration	Result
1. Simple pod restart	Original saved state (round 5,729,166, ~6 hours old)	Stuck in CHECKING at 10.1s
2. Copy fresh state from healthy peer	Node 1's latest state (round 5,824,918, <1 min old)	Stuck in CHECKING at 10.1s
3. Start with no state at all	All saved states and PCES removed — genesis start	Stuck in CHECKING at 10.1s

Key finding: The saved state age/content is irrelevant. The issue is at the gossip protocol level.

Successful Recovery

Deleting all 5 consensus node pods simultaneously forced all nodes to restart and enter CHECKING together. They collectively transitioned to ACTIVE within ~90 seconds. Zero ISS errors.

This is consistent with earlier observations: when all 5 nodes entered CHECKING simultaneously (due to correlated JVM pauses earlier the same day at 04:42-04:47 UTC), they recovered to ACTIVE within 1.2-1.4 minutes. Nodes in CHECKING can gossip with other CHECKING nodes, but not with ACTIVE nodes.

Expected Behavior

A single node in CHECKING should be able to receive gossip events from ACTIVE peers and eventually transition to ACTIVE. The gossip layer should not become completely dead (zero events, zero syncs) when peers are reachable and actively gossiping.

Analysis

The evidence points to a protocol-level deadlock:

1. CHECKING node needs events from peers to become ACTIVE. The 10.1-second OBSERVING → CHECKING transition suggests a timeout where the node expects to receive enough gossip to catch up.
2. ACTIVE nodes do not push events to CHECKING node (or the CHECKING node's gossip layer is not processing incoming connections). The zero-across-the-board gossip metrics confirm this.
3. When ALL nodes are in CHECKING, gossip works correctly. This was proven both by the 04:42-04:47 spontaneous recovery and the deliberate all-node restart at 10:47.

This creates a deadlock: the single CHECKING node waits for gossip that the ACTIVE nodes never send.

Open Questions

• Is this a known design limitation in v0.68.6 where ACTIVE nodes intentionally don't gossip with CHECKING nodes?
• Would a thread dump of the stuck node's JVM reveal blocked gossip threads?
• Is the 10.1-second OBSERVING timeout configurable? Would extending it help?
• Is there a fix or improvement planned for the reconnect path when a single node needs to rejoin an active network?

Impact

• Operational: Any single-node restart (planned maintenance, pod eviction, OOM recovery) can trigger this deadlock, requiring a full-network restart to recover.
• Staggered restart incompatibility: The recommended practice of restarting nodes one at a time for maintenance is fundamentally broken by this behavior — every staggered restart risks stranding the restarted node in CHECKING.
• Fault tolerance: While 4/5 nodes maintain consensus (80% weight > 2/3 threshold), the stranded node reduces fault tolerance from N-2 to N-3 until a disruptive full-network restart is performed.

Logs / Artifacts

Full investigation details, platform status timelines, gossip metric snapshots, and implementation log are available. Happy to provide thread dumps or additional diagnostics if guidance is provided on what would be most useful.

Start with the CHECKING transition timeline, the ReconnectController behavior, and the gossip metrics from the five-node reproduction. Investigate why a lone CHECKING node receives no gossip from ACTIVE peers, using the available logs and thread dumps if requested. Done means a staggered single-node restart can recover to ACTIVE without restarting the entire network.