[Bug] Parallel runner reports false "empty test suite" (exit 1 despite all tests passing) due to boolean-sum merge of numberOfTests

Open Beginner friendly
#1,738 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
php
Domain
testing-qa

Research direction

Start at src/Plugins/Parallel/Paratest/WrapperRunner.php:327 and inspect how worker TestResult instances are merged, including the ResultReflection number-of-tests access mentioned in the issue. Reproduce with vendor/bin/pest --parallel on a constrained two-core environment. Done means a passing suite reports that tests ran and exits 0 instead of being treated as empty.

Written by the indexing model from the issue text.

Description

When running tests in parallel (--parallel), the merged TestResult can report hasTests() === false even though hundreds of tests ran and passed. With PHPUnit 12.5.23+ (which defaults failOnEmptyTestSuite on), this makes the process exit 1 while the summary still shows e.g. 640 passed.

Environment
  • pest 4.7.4
  • phpunit/phpunit 12.5.30
  • brianium/paratest 7.20.0
  • PHP 8.5
Root cause

In src/Plugins/Parallel/Paratest/WrapperRunner.php:327, worker results are merged with:

(int) $testResultSum->hasTests() + (int) $testResult->hasTests(),

This passes a boolean sum as the TestResult $numberOfTests constructor argument instead of summing the real per-worker test counts. The per-worker numberOfTests is itself unreliable in the streaming worker (it does not reflect the tests actually run — numberOfTestsRun does). So the merged "has tests" flag becomes true only if at least one worker happens to report hasTests() === true.

Observed worker data (single shard, all tests passing)

2 workers:

worker_01  hasTests=false  numberOfTestsRun=401
worker_02  hasTests=false  numberOfTestsRun=239

18 workers (same suite):

worker_05  hasTests=true   numberOfTestsRun=12
worker_07  hasTests=true   numberOfTestsRun=40
... (13 others hasTests=false, e.g. numberOfTestsRun=401/239/43)

So the failure is process-count dependent: with few workers, all of them can report hasTests=false, making the merged result look empty; with many workers, at least one flukes to true and it passes. A 2-core CI runner deterministically fails; a many-core dev machine passes.

Steps to reproduce
  1. A test suite large enough to span multiple parallel workers, all passing.
  2. Constrain to 2 CPU cores (e.g. a 2-core CI runner, or shim the CPU finder).
  3. vendor/bin/pest --parallel → prints N passed, but exits 1.
Expected vs actual
  • Expected: exit 0 when all tests pass and tests were actually run.
  • Actual: exit 1, treated as an empty test suite.
Suggested fix

Sum the real test counts in the merge rather than the boolean flags, e.g.:

ResultReflection::numberOfTests($testResultSum) + ResultReflection::numberOfTests($testResult),
Workaround

Set failOnEmptyTestSuite="false" in phpunit.xml.

Dominant language
PHP
Stars
11.7k
Forks
538
Avg merge
4d 11h
Merged PRs (30d)
8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pestphp/pest

All issues in pestphp/pest

Similar issues

More PHP issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.