libp2p / libp2p/rust-libp2p

swarm: `Connection::poll` regularly exceeds Tokio's 50 µs slow-poll threshold

Open
#6,438 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
5.6k
Forks
1.3k
Avg merge
8h 47m
Merged PRs (30d)
19

Description

Context

Connection::poll (swarm/src/connection.rs:274) is a fixed-point loop
that re-polls every child sub-state-machine whenever any one returns
Ready, and only yields once every child returns Pending
simultaneously. With no upper bound on iterations per call, single
invocations regularly cross tokio's 50 µs slow-poll threshold
holding a tokio worker thread back from timers, the I/O reactor, and
colocated tasks for hundreds of microseconds at a time.

Findings

120-cell sweep (N ∈ {1,2,5,10,15} × RTT ∈ {0,5,25,50,100,200} ms × payload ∈ {4,16,64,256} KiB) — real TCP, per-peer netns + tc netem,
single-worker tokio runtime, taskset-pinned per process:

  • 106 / 120 cells have ≥ 1 % of Connection::poll invocations
    cross 50 µs. The 14 cells under 1 % are all at 4 KiB / N ≤ 2.
  • Worst cells (256 KiB / RTT=0 / N ≥ 2): 32–36 % of polls slow,
    p99 = 350–375 µs (~7× the threshold), and ~80 % of measurement
    wall-clock
    is spent inside slow polls.
  • Idle isn't clean either: 64 KiB / N=1 / RTT=0 = ~8 % slow ratio.
  • Swarm::poll is fine: peak 2.1 %, mean 0.22 %, under 1 % in 119/120
    cells. The cost is inside Connection::poll, not the event loop
    above it.

Reproducing

Full methodology, source, raw CSVs, and rendered heatmaps live in
latency-benchmark/.
One-liner: ./latency-benchmark/sweep.sh on Linux. sudo is only
required for tc/netns; RTT=0 cells skip both and run as a regular
user. ~50–60 min for the default grid.

Possible direction

A tokio-coop-style work budget on the inner loop — bound iterations
per Connection::poll invocation, self-wake on saturation — would cap
per-poll duration without changing observable behaviour beyond one extra
scheduling round per saturated poll. Sketched in
the bench README.
The specific budget value is empirical; we don't have a strong preference.

Happy to refine the methodology, add measurements, or produce
before/after numbers once a fix shape lands.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in swarm/src/connection.rs:274 to understand the fixed-point loop, then read latency-benchmark/README.md and run latency-benchmark/sweep.sh to establish the baseline. Define a bounded inner-loop budget with self-waking on saturation, and verify that the benchmark shows shorter polls without changing observable behavior beyond an extra scheduling round.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.