blockblaz / blockblaz/zig-libp2p
Peer book can drift from live conns: inbound-only peers not re-dialed; investigate silent conn loss with no lifecycle event
- Dominant language
- Zig
- Stars
- 5
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Context
A zeam node on devnet-eth2 wedged: head frozen 5+h, every `blocks_by_root`/`status` request returning `Disconnected` ("no conn to peer") and `IoError`, **0 `peer_connected` and 0 `dial` events in 10 minutes** despite the metric reporting **15 connected peers**, and `gossip_silent_ms=36006` with `mesh_peers=26` (36s of zero gossip). The transport-facing symptom is that zeam's peer view was decoupled from its actual live connections.
The primary root cause was on the zeam sync side (blockblaz/zeam#1046 — a false "peer does not support blocks_by_range" that degraded catch-up into a per-block storm). But the *trigger* underneath is here: connections went away and the app kept believing peers were connected, never re-establishing them.
## What's already correct (verified)
- `connection_manager.onConnectionClosed` emits `peer_disconnected`, calls `req_resp.onPeerDisconnected`, and **re-arms a redial for `known` peers** (retry-forever with capped backoff, `reconnectDelayMs`).
- QUIC transport params: `max_idle_timeout = 30s`, app-layer `/meshsub` keepalive = `20s` — healthy conns stay up; genuinely idle-dead conns should be reaped within 30s and flow through `detectOutboundConnectionClose` → `onConnectionClosed`.
So for **outbound / known** peers the loop is sound.
## Gaps to investigate
1. **Inbound-only peers are never re-dialed.** `onConnectionClosed` only re-arms a dial when `self.known.getPtr(peer)` hits. On a full mesh ~half of every node's peers are inbound-only (they dialed us; we never registered them as `known`). When such a conn dies, we emit `peer_disconnected` but never attempt to reconnect — and if the remote doesn't re-dial either, the peer is lost until restart. Consider: capture a dialable addr for inbound peers via identify/signed-peer-record and make them redial-eligible (guarded against NAT-undialable peers and against dial churn).
2. **Silent conn loss with no lifecycle event.** The 0-`peer_connected`/0-`dial` + stale-book + 36s gossip-silence pattern suggests either (a) the idle-timeout close wasn't surfaced for some conns, or (b) the drive loop / event propagation stalled while the by-root retry path kept running. Needs a live repro to distinguish; instrument `detectOutboundConnectionClose` / the inbound `syncSeenFlags` poll and the `peer_connected`/`peer_disconnected` emit counts vs. the conn_table size.
3. **Defense-in-depth (quinn/lighthouse parity):** inbound req/resp rate limiting so a storming peer (or a buggy remote sync loop like the one in #1046) is throttled rather than allowed to thrash connections.
## Why not patched blind here
The redial/idle/keepalive machinery is correct and this file set is heavily scarred by past dial-churn regressions (see the many "no eager redial / duplicate-connection churn" comments). A speculative change to inbound-peer redial or close-detection without a reproduction is likely to reintroduce churn. Filing to fix with a repro. The sync-layer amplifier (the thing that turned this into a multi-hour wedge) is fixed in blockblaz/zeam#1046.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with connection_manager.onConnectionClosed and trace detectOutboundConnectionClose, the inbound syncSeenFlags poll, peer lifecycle events, and conn_table updates. Reproduce the stale-peer condition while instrumenting close detection, event counts, and connection-table size to distinguish missing close events from a stalled drive loop. Done means the failure mode is identified and a scoped fix is validated without reintroducing dial churn.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- zig
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100