Nodes silently reject inbound libp2p connections from a specific access node until process restart

Summary

Over several days, an Access Node's ping engine failed to reach 6 nodes (4 consensus, 2 verification) operated by a partner. All failed with the same libp2p symptom: TCP connect succeeded, then the responder closed the connection before the Noise security handshake could complete. Throughout the entire period, the affected nodes were up and contributing to the chain as expected — the consensus nodes proposed, voted, and ran HotStuff normally, and network-level consensus finalization was not affected. The issue was strictly an inbound-libp2p rejection specific to this Access Node's peer id, not a node availability problem. All other nodes in the network were reachable from the Access Node. After the partner restarted the flow-go process on the 4 affected consensus nodes, they became reachable from the Access Node again with no configuration change. Root cause is unknown.

The fact that the symptom appeared on both consensus and verification nodes indicates the underlying cause is not role-specific.

Environment

• flow-go version on both sides: v0.51.0-rc.1
• Network: mainnet
• Reporting node: Access Node, node id 4e17496619df8bb4dcd579c252d9fb026e54995db0dc6825bdcd27bd3288a990
• Affected nodes (all partner-operated):
  • Consensus nodes:
    • 880b2a865c4169b378338eb69f49ebd835599399ebc061d0f960c082519622c5
    • 178e457f4512e0245e91f50705ab75ac4aaca400126192b7b39670b02679e53c
    • 154c77966417060df64207e1525cebd707457bfeec5e1c86df6b8d2982410501
    • cd7eed9952dbf8822062db7a351d959ef1166cd6d1856c0ad26c1c67d68e4fdb
  • Verification nodes:
    • 5f079d31810e4ea4ac86e70e23a04e797a37973df434d1ad854cd83d8828bcbd
    • 49c9f946170d5fb40c2af953b1534fae771905865b142ab6ac9685b8ba5b51c1

Symptom

The Access Node's ping engine (engine=ping) reported once per ping cycle for each affected node:

failed to ping peer <QmID>: failed to dial: failed to dial <QmID>: all dials failed
  * [/ip4/<ip>/tcp/3569] failed to negotiate security protocol: EOF

telnet <host> 3569 from the same source succeeded, confirming TCP-level reachability.

This is qualitatively distinct from the other failure modes observed in the same ping cycle for other unreachable peers:

• connection refused — TCP-level rejection
• no good addresses / no addresses — identity table has no valid multiaddr
• context deadline exceeded — timeout somewhere in dial/handshake
• peer id mismatch — Noise completed but static key hash didn't match expected id

failed to negotiate security protocol: EOF means the TCP handshake succeeded on the responder, then the responder closed the connection during (or immediately before) the libp2p Noise handshake.

Investigation

Ruled out

1. Version mismatch. Both sides ran the same binary (v0.51.0-rc.1), same libp2p security protocol ids.

2. Reporting AN's local peer-scoring state. flow-go's peer-scoring caches (network/p2p/scoring/internal/appSpecificScoreCache.go, network/p2p/cache/gossipsub_spam_records.go) are in-memory herocache. The Access Node was restarted multiple times, which wipes local scoring; failures persisted, so the AN's local score for these peers is not the cause.

3. flow-go gossipsub-layer gates. GossipSub peer scoring and peer gater operate at the pubsub layer, above libp2p connection setup. They cannot produce a mid-Noise close on the responder.

4. flow-go connection gater on the responder. network/p2p/connection/connection_gater.go: InterceptAccept is a no-op (:122), and InterceptSecured runs after Noise completes (:128). If the responder's flow-go gater were rejecting the AN, we would see a post-Noise close and a rejected inbound connection warn log with disallow_list_causes or filter details in the responder's log — not EOF during security protocol.

5. Partner-side node health / consensus participation / general infra. The affected nodes remained up and were contributing to the chain normally for the entire duration of the issue. Multiple independent lines of evidence confirm this:

   • Partner's operational view. The partner reported that the affected nodes were caught up, voting did not appear abnormal, and other consensus metrics did not look off. From the partner's monitoring perspective there was no signal that anything was wrong with these nodes; they explicitly noted they were not sure what they could have looked at to know a restart was needed.
   • Network-level finalization. Consensus finalization on the network was not impacted during the affected period. If 4 consensus nodes had actually been down or non-participating, finalization would have been visibly affected. It wasn't, which is strong external evidence that these nodes were actively participating in HotStuff throughout.
   • Direct log evidence. Consensus-engine logs from one of the affected consensus nodes during the failure window showed the node proposing blocks, casting votes, and running HotStuff without interruption — and, critically, accepting inbound libp2p connections from other Flow nodes on the same :3569 port at the same time. So the responder's libp2p stack, Noise implementation, and network path were fine for every peer except this specific Access Node.
   • Role-agnostic symptom. The same symptom appearing on verification nodes simultaneously rules out any consensus-role-specific cause.

Interpretation

The rejection was specific to the Access Node's peer id and happened at the libp2p layer between TCP accept and Noise completion. The plausible mechanisms that can close a connection in that narrow window on the responder side are:

• libp2p resource manager (rcmgr) hitting a per-peer or per-scope limit — closes post-accept, pre-security, presenting as EOF to the dialer.
• A lingering half-dead connection object / stuck goroutine tied to the AN's peer id, holding a per-peer rcmgr slot.
• OS-level stale TCP socket to that peer holding a per-peer resource that flow-go's rcmgr counts.

None of these mechanisms are specific to node role, which is consistent with both consensus and verification nodes being affected. None produce log lines at the default --loglevel=info, which made the issue effectively invisible from the responder's side — the operator had no local signal that any peer was being rejected.

Resolution

The partner restarted the flow-go process on the 4 affected consensus nodes. All 4 became reachable from the Access Node immediately after restart. No configuration change was made. Note that the restart interrupted only the nodes' networking — they had been contributing to the chain throughout, so this was a recovery of AN-visible reachability, not a recovery from a downed node.

Root cause

Unknown. The empirical fix (process restart) is consistent with stale in-memory state on the responder (rcmgr per-peer counter, lingering libp2p connection object, or stuck handshake goroutine), but no direct evidence was captured before restart to identify which.

Suggested follow-ups

1. Observability — the primary gap. From the partner's side there was no local signal that anything was wrong: internal metrics looked fine, consensus participation was normal, and the operator explicitly noted they had nothing to look at that would have suggested a restart was needed. flow-go should surface libp2p connection-level rejections (rcmgr blocks, per-peer/per-scope exhaustion, stuck-handshake situations) as first-class, INFO-level log lines with the offending peer id and the exhausted scope, and as metrics that can be alerted on. Without this, an operator running affected nodes has no way to detect that they are silently rejecting inbound connections from specific peers.

2. Diagnostic runbook. If this recurs, capture the following from the affected node before restarting:

   • Goroutine dump (SIGQUIT / runtime.Stack) — reveals stuck libp2p connection or handshake goroutines tied to a specific peer id.
   • libp2p_rcmgr_* metrics — a nonzero blocked_conn for a peer scope tied to the AN's peer id would confirm rcmgr-per-peer exhaustion.
   • ss -tnp for the pinging peer's source IP — reveals lingering half-dead TCP sockets.

3. Investigate whether libp2p rcmgr per-peer counters can decouple from actual live connections in flow-go's configuration (e.g., under kernel-level TCP resets that don't cleanly notify libp2p), and if so, whether a periodic reconciliation is warranted.

Start with the ping engine failure and network/p2p/connection/connection_gater.go, then inspect the libp2p resource-manager metrics and capture diagnostics before any restart. Compare responder-side logs, goroutine state, TCP sockets, and per-peer resource counters to determine the rejection mechanism. Done should include a reproducible cause and a clearly scoped observability or recovery change.