ConduitIO / ConduitIO/conduit

Flaky test suite: multiple order/timing-dependent failures make CI green unreliable

Open
#2,534 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

While merging #2533 (a `cmd/conduit`-only change), the `test` job failed **three consecutive reruns on three different, unrelated suites**, then passed on the fourth. All failing tests pass locally and none were touched by the PR — the full `-race ./...` suite has several independent flaky tests, so a fully-green run is a coin flip.

## Observed flakes (July 2026)
| Test | Package | Likely cause |
|---|---|---|
| `TestServiceLifecycle_Stop/user_stop:_forceful` | `pkg/lifecycle` | timing on forceful stop — **already tracked in #2521** |
| `TestClient_NotFound` / `TestClient_CacheMiss` / `TestClient_CacheHit` | `pkg/schemaregistry` | shared fake-registry / port or cache state under parallel load |
| `TestEncodeDecode_ExtractAndUploadSchemaStrategy` / `TestEncodeDecode_DownloadStrategy_Avro` | `pkg/plugin/processor/builtin/impl/avro/internal` | same schema-registry fixture contention |
| `TestRandOld` | `pkg/plugin/processor/builtin/internal/diff/lcs` | uses global `math/rand` (`rand.Seed(1)`), which other parallel tests share/perturb — seed no longer deterministic under `t.Parallel()` |

## Why it matters
"The test suite is the product's warranty." A suite that fails ~½ of clean runs trains maintainers to reflexively re-run red CI, which is exactly how a *real* regression slips through. It also burns CI minutes (4× here) and blocks solo-maintainer admin-merge (which correctly refuses to bypass a red check).

## Suggested fixes
- **`TestRandOld`**: give it a local `rand.New(rand.NewSource(seed))` instead of the global source, so parallel tests can't perturb its sequence. Cheap, deterministic.
- **schemaregistry / avro**: isolate the fake-registry per test (unique port / fresh cache), or drop `t.Parallel()` on the affected tests.
- **#2521 lifecycle**: track separately (existing).
- Consider a CI step that runs the suite with `-count=2` on a schedule to surface flakes deliberately, and quarantine known-flaky tests behind a build tag until fixed so they stop gating merges.

Not urgent, but it's eroding the value of the gate. Grouping the untracked ones here; #2521 covers the lifecycle case.

Contributor guide

Open the contributing guide

Research direction

Start by running the full `-race ./...` suite and inspect the named tests in `pkg/schemaregistry`, `pkg/plugin/processor/builtin/impl/avro/internal`, and `pkg/plugin/processor/builtin/internal/diff/lcs`. Reproduce the failures under parallel load, separating the lifecycle case already tracked in #2521. Done means the untracked tests remain reliable across repeated runs and no longer make clean CI runs intermittently fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.