sebastianbergmann / sebastianbergmann/phpunit

Option to suppress code coverage target validation

Open
#6,472 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature/code-coverage feature/configuration/cli status/waiting-for-feedback type/enhancement
Dominant language
PHP
Stars
20.1k
Forks
2.2k
Avg merge
7h
Merged PRs (30d)
61

Description

References: https://github.com/infection/infection/issues/2440 and https://github.com/infection/infection/pull/2848

Quick summary:

Infection is a mutation testing tool. It makes small changes to a project's source code (such as inverting the condition inside an if), runs the test suite and checks whether the tests still pass.
Before doing any of this, Infection needs to do an initial run of the test suite to gather coverage information. It does this by generating a new, temporary PHPUnit config (based on the project's config) and invoking PHPUnit.

Infection supports a --filter option that allows users to specifcy a list of files or directories to consider. It will only generate mutations for those files.
If this option is used, the generated PHPUnit config is modified to only list the relevant files or directories in <source><include>.

Problem:

If a project uses #[CoversClass()]/#[UsesClass()], PHPUnit (since version 12 I believe) will run some validation on those attributes in https://github.com/sebastianbergmann/phpunit/blob/197fc3445df847e7771bd8574e7f55d46fa1e44f/src/Runner/CodeCoverage.php#L228-L256
Because the full test suite is being run but the config <source> includes only a small subset of source files, an exception is thrown in https://github.com/sebastianbergmann/php-code-coverage/blob/3c97001bca11db4a3e6eee52f6e6924617364c3a/src/Target/Mapper.php#L82 and a "X is not a valid target for code coverage" warning is emitted.
This, combined with failOnPhpunitWarning="true" (the default and probably generally advisable setting), leads to a failed initial test run, which in turn leads Infection to abort its run.

Possible solution:

A number of options were explored in the issues I linked above. The TLDR is that this is difficult to properly handle on Infection's end.
The most straightforward solution would be to add a "don't validate code coverage targets" config option to PHPUnit.
Is that something PHPUnit would consider adding?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the validation flow in src/Runner/CodeCoverage.php and the referenced src/Target/Mapper.php path, then reproduce the failure with Infection's --filter behavior and failOnPhpunitWarning="true". Done means PHPUnit offers a configuration option that suppresses code coverage target validation for this case without changing the default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.