Flaky test suite: multiple order/timing-dependent failures make CI green unreliable
- Dominant language
- Go
- Stars
- 610
- Forks
- 63
- Avg merge
- 12h 28m
- Merged PRs (30d)
- 57
Description
While merging #2533 (a `cmd/conduit`-only change), the `test` job failed **three consecutive reruns on three different, unrelated suites**, then passed on the fourth. All failing tests pass locally and none were touched by the PR — the full `-race ./...` suite has several independent flaky tests, so a fully-green run is a coin flip.
## Observed flakes (July 2026)
| Test | Package | Likely cause |
|---|---|---|
| `TestServiceLifecycle_Stop/user_stop:_forceful` | `pkg/lifecycle` | timing on forceful stop — **already tracked in #2521** |
| `TestClient_NotFound` / `TestClient_CacheMiss` / `TestClient_CacheHit` | `pkg/schemaregistry` | shared fake-registry / port or cache state under parallel load |
| `TestEncodeDecode_ExtractAndUploadSchemaStrategy` / `TestEncodeDecode_DownloadStrategy_Avro` | `pkg/plugin/processor/builtin/impl/avro/internal` | same schema-registry fixture contention |
| `TestRandOld` | `pkg/plugin/processor/builtin/internal/diff/lcs` | uses global `math/rand` (`rand.Seed(1)`), which other parallel tests share/perturb — seed no longer deterministic under `t.Parallel()` |
## Why it matters
"The test suite is the product's warranty." A suite that fails ~½ of clean runs trains maintainers to reflexively re-run red CI, which is exactly how a *real* regression slips through. It also burns CI minutes (4× here) and blocks solo-maintainer admin-merge (which correctly refuses to bypass a red check).
## Suggested fixes
- **`TestRandOld`**: give it a local `rand.New(rand.NewSource(seed))` instead of the global source, so parallel tests can't perturb its sequence. Cheap, deterministic.
- **schemaregistry / avro**: isolate the fake-registry per test (unique port / fresh cache), or drop `t.Parallel()` on the affected tests.
- **#2521 lifecycle**: track separately (existing).
- Consider a CI step that runs the suite with `-count=2` on a schedule to surface flakes deliberately, and quarantine known-flaky tests behind a build tag until fixed so they stop gating merges.
Not urgent, but it's eroding the value of the gate. Grouping the untracked ones here; #2521 covers the lifecycle case.
Contributor guide
Research direction
Start by running the full `-race ./...` suite and inspect the named tests in `pkg/schemaregistry`, `pkg/plugin/processor/builtin/impl/avro/internal`, and `pkg/plugin/processor/builtin/internal/diff/lcs`. Reproduce the failures under parallel load, separating the lifecycle case already tracked in #2521. Done means the untracked tests remain reliable across repeated runs and no longer make clean CI runs intermittently fail.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100