[Bug] Parallel runner reports false "empty test suite" (exit 1 despite all tests passing) due to boolean-sum merge of numberOfTests
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- php
- Domain
- testing-qa
Research direction
Start at src/Plugins/Parallel/Paratest/WrapperRunner.php:327 and inspect how worker TestResult instances are merged, including the ResultReflection number-of-tests access mentioned in the issue. Reproduce with vendor/bin/pest --parallel on a constrained two-core environment. Done means a passing suite reports that tests ran and exits 0 instead of being treated as empty.
Written by the indexing model from the issue text.
Description
When running tests in parallel (--parallel), the merged TestResult can report hasTests() === false even though hundreds of tests ran and passed. With PHPUnit 12.5.23+ (which defaults failOnEmptyTestSuite on), this makes the process exit 1 while the summary still shows e.g. 640 passed.
Environment
- pest 4.7.4
- phpunit/phpunit 12.5.30
- brianium/paratest 7.20.0
- PHP 8.5
Root cause
In src/Plugins/Parallel/Paratest/WrapperRunner.php:327, worker results are merged with:
(int) $testResultSum->hasTests() + (int) $testResult->hasTests(),
This passes a boolean sum as the TestResult $numberOfTests constructor argument instead of summing the real per-worker test counts. The per-worker numberOfTests is itself unreliable in the streaming worker (it does not reflect the tests actually run — numberOfTestsRun does). So the merged "has tests" flag becomes true only if at least one worker happens to report hasTests() === true.
Observed worker data (single shard, all tests passing)
2 workers:
worker_01 hasTests=false numberOfTestsRun=401
worker_02 hasTests=false numberOfTestsRun=239
18 workers (same suite):
worker_05 hasTests=true numberOfTestsRun=12
worker_07 hasTests=true numberOfTestsRun=40
... (13 others hasTests=false, e.g. numberOfTestsRun=401/239/43)
So the failure is process-count dependent: with few workers, all of them can report hasTests=false, making the merged result look empty; with many workers, at least one flukes to true and it passes. A 2-core CI runner deterministically fails; a many-core dev machine passes.
Steps to reproduce
- A test suite large enough to span multiple parallel workers, all passing.
- Constrain to 2 CPU cores (e.g. a 2-core CI runner, or shim the CPU finder).
vendor/bin/pest --parallel→ printsN passed, but exits1.
Expected vs actual
- Expected: exit
0when all tests pass and tests were actually run. - Actual: exit
1, treated as an empty test suite.
Suggested fix
Sum the real test counts in the merge rather than the boolean flags, e.g.:
ResultReflection::numberOfTests($testResultSum) + ResultReflection::numberOfTests($testResult),
Workaround
Set failOnEmptyTestSuite="false" in phpunit.xml.
- Dominant language
- PHP
- Stars
- 11.7k
- Forks
- 538
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 8
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from pestphp/pest
-
[Bug]: [mutate] Windows: @pest-mutate-ignore comments land on line 1 in files with LF line endings Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
getgrav/grav-plugin-api#45 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
RSS-Bridge/rss-bridge#5098 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
phingofficial/phing#2025 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
silverstripe/developer-docs#911 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100