glideapps / glideapps/quicktype
CI structure amplifies flakes: fail-fast matrix + test-complete mirror check
- Dominant language
- TypeScript
- Stars
- 13.9k
- Forks
- 1.2k
- Avg merge
- 8h 53m
- Merged PRs (30d)
- 369
Description
## What happens
Two aspects of the fixture-test workflow (`.github/workflows/test-pr.yaml`) inflate the blast radius of every flaky job and hide the results of unrelated languages.
1. **`fail-fast: true` on the fixture matrix.** The `test` job's matrix (`strategy.fail-fast: true`, line 34) runs every language fixture as a matrix leg. When any one leg fails — including a pure infra flake — GitHub cancels all sibling legs. A single flaky job therefore turns into a wall of "cancelled" jobs, and we lose the signal of whether the other languages would have passed.
2. **`test-complete` only mirrors the matrix.** The `test-complete` job (line 211) runs `if: cancelled() || contains(needs.*.result, 'cancelled') || contains(needs.*.result, 'failure')` and simply prints "Some workflows have failed!" and `exit 1`. It never fails independently — it only reflects whatever the matrix did. During triage its failure carries no information beyond "some matrix leg failed or was cancelled."
## Evidence
- `.github/workflows/test-pr.yaml` line 34: `fail-fast: true` on the `test` matrix.
- `.github/workflows/test-pr.yaml` lines 211-218: `test-complete` gate that mirrors the matrix result.
- Today's triage of 31 failing PRs: several PRs showed many "cancelled" sibling jobs stemming from a single flaky leg, and `test-complete` "Some workflows have failed!" was never an independent failure — it only mirrored whichever matrix job failed.
## Impact
- One flaky job (Scala Bloop timeout, Elm warmup deadlock, cjson download reset) cancels the entire language matrix.
- Triage is harder: the real failing leg is buried among cancellations, and passing languages never report.
- The `test-complete` red X is noise on its own; you must open the matrix to learn anything.
## Suggested solutions
- Set `fail-fast: false` on the fixture matrix so each language leg runs to completion independently and one flake stops masking the rest.
- During triage, treat `test-complete` as a mirror only — inspect the individual matrix legs, not the aggregate gate.
## Related flaky areas
These are the specific flakes this structure amplifies:
- #3063 — scala3-upickle: Bloop startup timeout + maven-nightlies download failures
- #3064 — elm: Warmup.elm setup races the compiler (file locks / MVar deadlock)
- #3065 — cjson: cJSON.c downloaded from raw.githubusercontent.com at test time
## Interim policy
Per `CLAUDE.md` ("Known CI flakiness"), we accept these flakes for now and retry the failed jobs (`gh run rerun --failed`) when they occur.
Contributor guide
Research direction
Start in .github/workflows/test-pr.yaml at the test matrix around line 34 and the test-complete gate around lines 211-218. Review how matrix results are propagated, then verify that one failed or flaky language leg no longer cancels sibling legs and that the aggregate status remains understandable during triage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- ci-cd
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100