pestphp / pestphp/pest

[Bug]: [mutate] Windows: quoted --filter breaks cmd when a line is covered by several tests, so every mutant counts as killed

Open Beginner friendly
#1,927 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
11.7k
Forks
538
Avg merge
4d 11h
Merged PRs (30d)
8

Description

What Happened

On Windows, pest --mutate reports every mutant as killed (score 100%) whenever the mutated line is covered by more than one test. This happens even when the per-mutant child process launches correctly, so it's separate from the forward-slash argv[0] problem in #1916.

MutationTest::start() wraps the filter in literal quotes:

'--filter="'.implode('|', $filters).'"',

The command is an array, so Symfony Process already escapes each argument. On Windows it doubles the embedded ", which ends the quoted section for cmd.exe, so the | between two filters is parsed as a pipe:

'TtyTest::' is not recognized as an internal or external command,
operable program or batch file.

The child exits non-zero, hasFinished() maps that to Tested, and the mutant counts as killed.

Expected: the child runs the covering tests, and a mutant that no test kills is reported as untested.

How to Reproduce

Isolated with the child command built the way the plugin builds it. argv[0] is the absolute path Composer's .bat proxy passes, which launches fine:

use Symfony\Component\Process\Process;

$filter = 'TtyTest::(.*)it.runs.a.binary.from.bin.dir|TtyTest::(.*)it.passes.the.exit.code.through';

$process = new Process([getcwd().'\vendor\bin\/pest', '--bail', '--filter="'.$filter.'"']);
$process->run();
// exit 255: 'TtyTest::' is not recognized as an internal or external command

$process = new Process([getcwd().'\vendor\bin\/pest', '--bail', '--filter='.$filter]);
$process->run();
// exit 0: the 4 matching tests run and pass

End to end, with the sample repository on Windows (PHP 8.5 with Xdebug):

  1. composer install
  2. In src/Tty.php, delete the two // @pest-mutate-ignore: RemoveArrayItem comments. That leaves 2 mutants no test can kill.
  3. composer test:mutate reports 100%. On Linux, the same run reports the 2 survivors (94.87%).

Suggested fix: drop the literal quotes, '--filter='.implode('|', $filters),. The process escapes the argument itself.

Sample Repository

https://github.com/Shoxcie/composer-tty

Pest Version

5.2.1 (pestphp/pest-plugin-mutate v5.0.2, symfony/process v8.1.7)

PHP Version

8.5.10

Operation System

Windows

Notes

Filed here because issues are disabled on pestphp/pest-plugin-mutate.

Related: #1916 (forward-slash argv[0]) and #1313 (Windows absolute paths in <source>). This bug remains after working around both. The sample repository's test:mutate runs through Composer, so argv[0] launches, and it passes --path=src, so the files are found. Yet every mutant still counts as killed. As #1916 points out, the deeper problem is that any failed child run counts as a kill, so a broken command silently produces a perfect score.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at MutationTest::start(), where the mutation command builds its --filter argument, and compare the quoted and unquoted Windows command forms shown in the issue. Run the provided composer-tty reproduction on Windows; done means covering tests execute successfully and mutants that no test kills are reported as survivors rather than all being counted as killed.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, symfony
Domain
cli, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
80/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.