blockblaz / blockblaz/zig-libp2p
quic: residual handshake stalls + ACK-starvation under 3-node all-to-all load (post-#245)
- Dominant language
- Zig
- Stars
- 5
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Follow-up to #245 (non-blocking dials), which fixed the **permanent** Initial-handshake deadlock that forked zeam devnets. After that fix a 3-node zeam devnet on a single box now **justifies steadily** (slot 0→18 in lockstep with wall-clock; previously stuck at finalized=slot 0). But residual transport churn remains and should be chased down separately.
## Observed (zeam `beam --is-aggregator true`, 3 in-process nodes, all-to-all `quic-v1`, single box)
Over a 90s run:
- ~7× `quic_runtime: dial handshake timed out after 20000ms; … stalled_phase=initial` — recurring to **2 specific peers**, throughout the run (not just boot).
- ~4× `zquic: io: client declaring connection lost (no ACK for 60s …); marking draining`, one with `bif=2497811, ld=2048/2048, pending=5` — i.e. ~2.4 MB in flight, loss-detector tracking array fully saturated, **zero ACKs for 60s** on an established connection, then teardown.
Consensus stays live (a stable 2/3 supermajority keeps justifying), so this is degraded-but-live, not a hard fork.
## Hypotheses to investigate
1. **ACK generation / pacing under load** — `ld=2048/2048` with zero ACKs for 60s suggests the receiver side isn't ACKing (or the sender isn't processing ACKs). Is the receiver's `driveLoop` servicing inbound packets + emitting ACKs promptly when also doing gossip/req-resp/identify work each tick?
2. **CPU starvation of the drive thread** — zeam aggregator STARK/XMSS proving is multi-second and CPU-heavy; 3 full nodes + provers on one box may starve the QUIC drive thread enough to miss handshake/ACK deadlines. Need to repro on separate hosts / with proving disabled to isolate transport from environment.
3. **`advancePendingDials` services pending outbound sockets with `drive(buf, 0)` (non-blocking) once per tick**, gated by the listener's 5ms poll — confirm that's frequent enough for a handshake under load, or whether pending dials need their own poll budget.
4. Congestion controller behaviour when ACKs are absent — does cwnd/bytes-in-flight back off, and is the 60s idle teardown the right signal vs. earlier PTO-driven retransmit give-up?
## Repro
`ssh` box, `cd /mnt/data/zeam_repo` (on `feat/replace-libp2p-glue`, dep = v0.1.98), `./zig-out/bin/zeam beam --is-aggregator true`; watch stderr for `stalled_phase` / `marking draining`, and poll `http://localhost:9667/lean/v0/checkpoints/justified` for liveness.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the 3-node all-to-all run from the provided zeam command and inspect the stalled_phase and marking draining logs. Start with driveLoop, advancePendingDials, the pending-socket drive(buf, 0) path, and the listener's 5ms poll, then compare runs with proving disabled or on separate hosts. Done means identifying the transport cause and confirming that handshake and ACK stalls no longer recur while checkpoint justification remains live.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- zig
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100