pestphp / pestphp/pest

[Bug]: Using `--parallel` causes overall process to fail with no reporting as to why even though all tests pass

Open
#1,483 14 comments 11 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
PHP
Stars
11.7k
Forks
538
Avg merge
4d 11h
Merged PRs (30d)
8

Description

What Happened

Our team noticed after upgrading to Pest 4 that our tests running in parallel are showing all tests as passing, but having the overall command fail. However, if we run the exact same test suite sequentially, we get the same number of passing tests and the overall command passes.

Sequential

./vendor/bin/pest --exclude-group=fails-in-parallel  --testsuite=Feature --compact

  ................................................................................................................................................................
  ............................................................................

  Tests:    236 passed (364 assertions)
  Duration: 139.51s

echo $?
0

Parallel

./vendor/bin/pest --exclude-group=fails-in-parallel  --testsuite=Feature --parallel

  ................................................................................................................................................................
  ............................................................................

  Tests:    236 passed (364 assertions)
  Duration: 30.20s
  Parallel: 12 processes

echo $?
2
How to Reproduce
  1. Run test suite without --parallel flag and see it passes with the same number of tests and assertions
  2. Run test suite with --parallel flag and see it returns an error code of 2, even though it shows the same number of tests and assertions passing successfully
Sample Repository

No response

Pest Version

4.0.3

PHP Version

8.4.11

Operation System

macOS

Notes

I'm not certain if it's related, but I've also noticed that when we run tests in parallel now and there is a test failure, it is swallowing the error and immediately killing the tests without reporting anything about what caused the failure. There was a previous issue #1451 that hinted at their problem could be related to the master process not properly handling the exit codes of the worker threads, which feels like it could be a reason for both of these issues.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Pest's --parallel execution path and the master process's handling of worker exit codes; issue #1451 is the related report mentioned in the issue. Reproduce with the sequential and parallel commands against Pest 4.0.3, then verify that parallel failures and nonzero exits report their cause instead of silently terminating.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
cli, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.