[Bug]: Pest 5 concurrency — parallel coverage loses a covered line, and --mutate --parallel reports survivors as killed
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 11.7k
- Forks
- 538
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 8
Description
What Happened
Three independent findings landed on the same day across a 46-repository fleet on Pest 5. Individually each looked like bad luck. Together they are one shape: a Pest 5 run that forks produces a different, quieter answer than the same run serial — and in two of the three the wrong answer is the more flattering one.
I am filing them together deliberately, because the pattern is the point. Each is separable and carries its own evidence.
One of the three has since been confirmed and fixed by a maintainer, which is why I think the other two are worth a look rather than a shrug.
1 — A parallel coverage run loses a line that is demonstrably covered
Status: reproducible in place, deterministic, not yet minimised.
| serial | pest --coverage --min=100 --exclude-group=browser → 100.0% |
| parallel | pest --coverage --min=100 --exclude-group=browser --parallel --processes=4 → 99.9% |
| the line | EmailTwoFactorChallengeController line 197, reported uncovered |
| repeats | same line, every parallel run |
| control | on Pest 4, this exact parallel command was green |
Same tree, same tests, same driver settings (PCOV, XDEBUG_MODE=off exported as an env var so workers inherit it — not -d). The only variable is --parallel, and php-code-coverage moving 12.5.7 → 14.2.3 with the major.
It is not a threshold artefact: serial says the line is covered, parallel says it is not, and it picks the same line each time. That reads like a coverage merge losing one worker's contribution rather than a flaky collection.
The workaround in place is to run coverage serially, which costs wall-clock and nothing else — but it means every repository that trusts --parallel --coverage on Pest 5 may be enforcing a floor against a number that is slightly too low, and would fail a --min=100 gate for a line that is fine.
I have not minimised this to a standalone package yet. If a smaller reproducer is what unblocks it, say so and I will build one — I did not want to sit on the field evidence while doing that.
2 — --mutate --parallel reports surviving mutants as killed
Status: hand-verified, and the wrong number is the higher one.
Same class, same tests, same command apart from --parallel:
| MSI | |
|---|---|
pest --mutate (serial) |
89% |
pest --mutate --parallel |
100% |
The survivors that vanish under --parallel are real: verified by hand against the serial run. So the parallel score is not noisier — it is inflated, in the direction nobody double-checks. A score that reads 100% is the one least likely to be questioned.
This is now a standing rule across the fleet: mutation runs serially, always, and parallelism is expressed as sharding at the container level (one serial --mutate per shard) rather than --parallel. That is a workaround for consumers; upstream it seems worth either fixing or refusing the flag combination outright, because the failure mode is a fabricated pass.
This is adjacent to #1316 but not the same direction: there, tested code is reported UNTESTED; here, untested code is reported killed.
3 — The type-coverage cache rewrote itself (already fixed — included as the confirmation)
pest --type-coverage --min=100, no --parallel, died with:
ParseError: syntax error, unexpected single-quoted string " => "
at vendor/pestphp/pest-plugin-type-coverage/.temp/v3.php:14
12▕ );baef33c5698f9926572068f087c' =>
13▕ array (
➜ 14▕ 0 => '/…/app/Models/SystemFile.php',
The cache file is torn mid array-literal: two writes interleaved. Stack trace named Pokio\Runtime\Fork\ThrowableCapsule::__unserialize(). Turning concurrency off at the command line does not escape it, because the plugin forks internally. And it is a race — the next run over the identical tree was green.
This one is already fixed, in pest-plugin-type-coverage v5.0.1 (fix: parallel corruption file). The old withinLock() tried flock(LOCK_EX|LOCK_NB) a hundred times over 100 ms and then, on giving up, ran the callback anyway — an unlocked write. Making the lock blocking closed it.
I include it because it is the same family and it was real, so it is some evidence that 1 and 2 are worth measuring rather than dismissing as environment.
How to reproduce
For 1 and 2: any suite large enough to have a coverage merge worth doing. Run each command twice, once with --parallel and once without, and diff the reports — the difference is stable, not flaky.
Sample Code / Versions
Pest 5, php-code-coverage 14.2.3, PCOV, PHP 8.4/8.5, Linux containers (4 processes) and macOS both. Finding 3 on pest-plugin-type-coverage v5.0.0, fixed in v5.0.1.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the listed serial and parallel coverage and mutation commands on a representative suite, then diff their reports across repeated runs. Focus on the two unresolved findings, using the stated Pest, php-code-coverage, PHP, and PCOV versions; done means parallel results agree with serial results or the unsupported combination is clearly handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100