blockblaz / blockblaz/zig-libp2p

quic: residual handshake stalls + ACK-starvation under 3-node all-to-all load (post-#245)

Open
#246 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Zig
Stars
5
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Follow-up to #245 (non-blocking dials), which fixed the **permanent** Initial-handshake deadlock that forked zeam devnets. After that fix a 3-node zeam devnet on a single box now **justifies steadily** (slot 0→18 in lockstep with wall-clock; previously stuck at finalized=slot 0). But residual transport churn remains and should be chased down separately.

## Observed (zeam `beam --is-aggregator true`, 3 in-process nodes, all-to-all `quic-v1`, single box)
Over a 90s run:
- ~7× `quic_runtime: dial handshake timed out after 20000ms; … stalled_phase=initial` — recurring to **2 specific peers**, throughout the run (not just boot).
- ~4× `zquic: io: client declaring connection lost (no ACK for 60s …); marking draining`, one with `bif=2497811, ld=2048/2048, pending=5` — i.e. ~2.4 MB in flight, loss-detector tracking array fully saturated, **zero ACKs for 60s** on an established connection, then teardown.

Consensus stays live (a stable 2/3 supermajority keeps justifying), so this is degraded-but-live, not a hard fork.

## Hypotheses to investigate
1. **ACK generation / pacing under load** — `ld=2048/2048` with zero ACKs for 60s suggests the receiver side isn't ACKing (or the sender isn't processing ACKs). Is the receiver's `driveLoop` servicing inbound packets + emitting ACKs promptly when also doing gossip/req-resp/identify work each tick?
2. **CPU starvation of the drive thread** — zeam aggregator STARK/XMSS proving is multi-second and CPU-heavy; 3 full nodes + provers on one box may starve the QUIC drive thread enough to miss handshake/ACK deadlines. Need to repro on separate hosts / with proving disabled to isolate transport from environment.
3. **`advancePendingDials` services pending outbound sockets with `drive(buf, 0)` (non-blocking) once per tick**, gated by the listener's 5ms poll — confirm that's frequent enough for a handshake under load, or whether pending dials need their own poll budget.
4. Congestion controller behaviour when ACKs are absent — does cwnd/bytes-in-flight back off, and is the 60s idle teardown the right signal vs. earlier PTO-driven retransmit give-up?

## Repro
`ssh` box, `cd /mnt/data/zeam_repo` (on `feat/replace-libp2p-glue`, dep = v0.1.98), `./zig-out/bin/zeam beam --is-aggregator true`; watch stderr for `stalled_phase` / `marking draining`, and poll `http://localhost:9667/lean/v0/checkpoints/justified` for liveness.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the 3-node all-to-all run from the provided zeam command and inspect the stalled_phase and marking draining logs. Start with driveLoop, advancePendingDials, the pending-socket drive(buf, 0) path, and the listener's 5ms poll, then compare runs with proving disabled or on separate hosts. Done means identifying the transport cause and confirming that handshake and ACK stalls no longer recur while checkpoint justification remains live.

Written by the indexing model from the issue text.

Assessment

Tech stack
zig
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.