blockblaz / blockblaz/zig-libp2p

Flaky CI: "Sustained gossipsub soak test (#235)" intermittently deadlocks harness teardown → job hangs for hours

Open
#270 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Zig
Stars
5
Forks
1
PR merge metrics
No merged PRs in 30d

Description

The `Sustained gossipsub soak test` step in the `zig` CI job intermittently **hangs indefinitely** (observed 90+ minutes) instead of its normal duration.

### Evidence
- The normal `zig` job (which includes this soak step) completes in **~7 minutes** (e.g. runs at 09:22→09:28, 08:18→08:24, 22:51→22:57 on `main`).
- On PR #269 the run started 10:25Z and was still stuck on the step `Sustained gossipsub soak test (#235)` **90+ minutes later**; cancelling + re-triggering the same commit passed cleanly in ~7 min. So it is **flaky/non-deterministic**, not a code defect in that PR.

### Root cause (self-documented)
The test `QuicRuntime: gossip saturation race repro` (src/transport/quic/runtime.zig) carries this comment:

> Soak-gated: a deliberately extreme multi-shard burst (heavier than live) used to REPRODUCE the advanceInboundStreams ist.conn UAF. It is heavy (~50s) and **can flakily deadlock the harness teardown** under the synthetic overload, so it is excluded from normal CI. Run with `-Denable-soak-tests`.

So the test itself acknowledges a flaky **teardown deadlock** under the synthetic overload. It is excluded from `zig build test`, but a dedicated CI step runs it with `-Denable-soak-tests`, where the deadlock surfaces and hangs the job (no per-step timeout → the run sits until the 6h GitHub default).

### Impact
- Blocks merges (PR sits `UNSTABLE` for hours).
- Wastes a CI runner for hours per occurrence.
- Trains reviewers to ignore/force-merge, eroding the signal.

### Suggested remediation (any of)
1. **Fix the teardown deadlock** in the soak harness (the cluster drainer thread join / shutdown ordering under sustained overload) so teardown is deterministic — this is the real fix; the test is valuable.
2. Add a **hard per-step timeout** (e.g. `timeout-minutes: 10`) on the soak CI step so a hang fails fast instead of hanging for hours, plus auto-retry-once.
3. Add an **internal watchdog** to the soak test: if the burst/teardown does not complete within N×expected, abort with a clear diagnostic instead of deadlocking.

Until fixed, the practical workaround is cancel + re-trigger (it passes ~most runs).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `QuicRuntime: gossip saturation race repro` test and its comment in `src/transport/quic/runtime.zig`, then inspect the CI definition for the `zig` job's `Sustained gossipsub soak test (#235)` step. Determine whether teardown ordering, a watchdog, or a per-step timeout best prevents indefinite hangs; done means soak runs either complete deterministically or fail promptly with a useful diagnostic.

Written by the indexing model from the issue text.

Assessment

Tech stack
zig
Domain
ci-cd, networking, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.