blockblaz / blockblaz/zig-libp2p
Flaky CI: "Sustained gossipsub soak test (#235)" intermittently deadlocks harness teardown → job hangs for hours
- Dominant language
- Zig
- Stars
- 5
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
The `Sustained gossipsub soak test` step in the `zig` CI job intermittently **hangs indefinitely** (observed 90+ minutes) instead of its normal duration.
### Evidence
- The normal `zig` job (which includes this soak step) completes in **~7 minutes** (e.g. runs at 09:22→09:28, 08:18→08:24, 22:51→22:57 on `main`).
- On PR #269 the run started 10:25Z and was still stuck on the step `Sustained gossipsub soak test (#235)` **90+ minutes later**; cancelling + re-triggering the same commit passed cleanly in ~7 min. So it is **flaky/non-deterministic**, not a code defect in that PR.
### Root cause (self-documented)
The test `QuicRuntime: gossip saturation race repro` (src/transport/quic/runtime.zig) carries this comment:
> Soak-gated: a deliberately extreme multi-shard burst (heavier than live) used to REPRODUCE the advanceInboundStreams ist.conn UAF. It is heavy (~50s) and **can flakily deadlock the harness teardown** under the synthetic overload, so it is excluded from normal CI. Run with `-Denable-soak-tests`.
So the test itself acknowledges a flaky **teardown deadlock** under the synthetic overload. It is excluded from `zig build test`, but a dedicated CI step runs it with `-Denable-soak-tests`, where the deadlock surfaces and hangs the job (no per-step timeout → the run sits until the 6h GitHub default).
### Impact
- Blocks merges (PR sits `UNSTABLE` for hours).
- Wastes a CI runner for hours per occurrence.
- Trains reviewers to ignore/force-merge, eroding the signal.
### Suggested remediation (any of)
1. **Fix the teardown deadlock** in the soak harness (the cluster drainer thread join / shutdown ordering under sustained overload) so teardown is deterministic — this is the real fix; the test is valuable.
2. Add a **hard per-step timeout** (e.g. `timeout-minutes: 10`) on the soak CI step so a hang fails fast instead of hanging for hours, plus auto-retry-once.
3. Add an **internal watchdog** to the soak test: if the burst/teardown does not complete within N×expected, abort with a clear diagnostic instead of deadlocking.
Until fixed, the practical workaround is cancel + re-trigger (it passes ~most runs).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the `QuicRuntime: gossip saturation race repro` test and its comment in `src/transport/quic/runtime.zig`, then inspect the CI definition for the `zig` job's `Sustained gossipsub soak test (#235)` step. Determine whether teardown ordering, a watchdog, or a per-step timeout best prevents indefinite hangs; done means soak runs either complete deterministically or fail promptly with a useful diagnostic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- zig
- Domain
- ci-cd, networking, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100