pestphp / pestphp/pest

[mutate] Windows: forward-slash outer argv[0] fails to launch per-mutant children, every mutant silently reports killed

Open
#1,916 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
11.7k
Forks
538
Avg merge
4d 11h
Merged PRs (30d)
8

Description

Correction, see the comment below for the full account: the mechanism this report originally described — CliConfiguration.php:64's hardcoded literal reaching the per-mutant child — is wrong. That literal is local to fromArguments(), discarded after building an ArgvInput for option parsing, and never returned. The per-mutant child's command is actually built from the outer --mutate process's own argv[0] (via MutationTestRunner::setOriginalArguments()), so the fix is at the call site (e.g. an @php-prefixed Composer script on Windows, which normalizes the separator), not in this project. The reproduction, symptom, and impact described below are all still accurate; only the attributed cause is not.

Filing here because issues are disabled on pestphp/pest-plugin-mutate. Related but distinct from #1843, which is a different root cause (a Symfony Process deadlock inside a mutant's own test) reaching the same class of symptom — any non-isSuccessful() process outcome gets scored as a kill rather than reported as its own state.

What:
  • Bug Fix
Description:

On Windows, CliConfiguration::fromArguments() hardcodes the per-mutant child process's argv[0] as the literal forward-slash string 'vendor/bin/pest':

// src/Support/Configuration/CliConfiguration.php:64
$filteredArguments = ['vendor/bin/pest'];

This string is reused verbatim in MutationTest::start() to build every per-mutant Process:

// src/MutationTest.php:85-93
$process = new Process(
    command: [
        ...$originalArguments,   // starts with the hardcoded 'vendor/bin/pest'
        '--bail',
        '--filter="'.implode('|', $filters).'"',
    ],
    env: $envs,
    timeout: $this->calculateTimeout(),
);

On Windows, a forward-slash relative path in the first element of a Symfony Process array command fails to launch. Isolated in a minimal reproduction against symfony/process 7.4.13, same working directory, only the separator and extension varied:

'vendor/bin/pest'          exit 1, stderr: 'vendor' is not recognized as an internal or external command, operable program or batch file.
'vendor\bin\pest'          exit 0, runs correctly
'vendor/bin/pest.bat'      exit 1, same error
'vendor\bin\pest.bat'      exit 0, runs correctly

Only the separator matters; the .bat extension does not change the outcome either way.

Because MutationTest::hasFinished() reads $this->process->isSuccessful() and maps a non-zero exit to MutationTestResult::Tested (killed) rather than treating a launch failure as its own state, every mutant whose covering-test child fails to launch this way is reported as killed, indistinguishable from a genuine kill:

// src/MutationTest.php:139-150
if ($this->process->isSuccessful()) {
    $this->updateResult(MutationTestResult::Untested);
    // ...
    return true;
}

$this->updateResult(MutationTestResult::Tested);

Reproduction

On Windows, running --mutate --covered-only --path=<any file> against a real project: every mutant reports as tested/killed, with a score of 100%, and real (non-instant) per-mutant wall-clock time — because each child process takes roughly 1.5-2s to fail to launch and report the error, which reads as plausible execution time. Confirmed by disabling a test known to kill a specific mutant and re-running: the mutant still reports killed, because the process that should have run that test never launched at all.

Instrumented MutationTest.php directly (two fwrite(STDERR, ...) calls printing the constructed command, exit code, and isSuccessful()) and ran a 38-mutant pass on one file: all 38 child processes failed to launch ('vendor' is not recognized...), all 38 were marked killed, reported score 100.00%.

Impact

This is not a narrow edge case — it affects every --mutate invocation on Windows, regardless of how the outer pest --mutate ... command was invoked (bare, absolute path, .bat, through composer exec, through a composer.json script). The outer invocation's form is irrelevant, because the failing command is built from the hardcoded literal in CliConfiguration::fromArguments(), not from the outer process's own argv. On an affected machine, mutation testing produces a plausible-looking, fully fabricated 100% for every run, with no error and no indication anything is wrong.

Suggested fix (superseded, see correction comment)

The CliConfiguration.php change originally proposed here would not fix anything, since that code path is never spawned. What would help: MutationTest::start() normalizes $originalArguments[0]'s path separator (DIRECTORY_SEPARATOR, or Symfony\Component\Filesystem\Path::normalize()) before building the per-mutant Process, so the child's launch does not depend on how the outer process happened to be invoked.

Environment

  • pestphp/pest-plugin-mutate v3.0.5
  • symfony/process 7.4.13
  • Windows 11, Git Bash / MSYS2, PHP 8.4.20 (Herd)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with MutationTestRunner::setOriginalArguments() and MutationTest::start() to trace how the outer process arguments become each child command; review the Windows reproduction using --mutate --covered-only --path=. Done means per-mutant children launch on Windows and a launch failure is no longer reported as a genuine killed mutant.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, symfony
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.