Mutation testing: --filter exceeds the kernel's per-argument limit (and PCRE's compile limit) on widely-covered classes
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 11.7k
- Forks
- 538
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 8
Description
Filed here because issues are disabled on
pestphp/pest-plugin-mutate. The code
referenced below lives in that package.
What happens
On a class that many tests cover, every mutation dies before it runs:
proc_open(): posix_spawn() failed: Argument list too long
at vendor/symfony/process/Process.php:411
The mutation is reported as not measured, and with a shard-per-directory CI setup the
whole run fails.
Why
MutationTest::start() selects the covering tests by building one regex fragment per test
and joining all of them into a single argv element:
// pest-plugin-mutate/src/MutationTest.php:92
'--filter="'.implode('|', $filters).'"',
Linux caps one argv element at MAX_ARG_STRLEN = PAGE_SIZE * 32
(include/uapi/linux/binfmts.h). x86_64 always has 4 KiB pages, so the cap is exactly
131072 bytes, and it cannot be tuned — ulimit raises the total ARG_MAX, never the
per-element limit. macOS has no per-element cap at all, which is why this reproduces in CI
and not on a developer machine.
Measured on one real mutation, in a small utility class that essentially every component
test reaches (2076 covering tests across 179 classes):
| encoding | bytes | vs 131072 |
|---|---|---|
| one fragment per test (current) | 172,779 | over |
| factor the class prefix | 124,987 | fits, +5% only |
collapse each class to Cls:: |
4,205 | fits, 31x headroom |
It is over two limits, not one
Worth knowing before picking a fix: PCRE also refuses that pattern.
preg_match(): Compilation failed: regular expression is too large at offset 163235
So even a transport that allowed a 170 KB argument would hand PHPUnit a filter it cannot
compile. The encoding itself has to get shorter; moving the same payload elsewhere is not
enough.
The TODO above the loop cannot fix it as written
// TODO: we should pass the tests to run in another way, maybe via cache, mutation or env variable
A cache or file carrier works. An environment variable does not:
do_execveat_common() runs the same copy_strings() over envp as over argv, and the
MAX_ARG_STRLEN check lives inside it — the environment is under the identical
per-element cap. Flagging it only so that route is not chosen and found wanting later.
What we did, and would gladly contribute
An adaptive encoding, applied in order:
- Factor the class prefix — always.
Cls::(.*)a|Cls::(.*)bbecomesCls::(.*)(a|b).
Lossless: replayed against the real test list, both forms select the identical set. The
inner parentheses matter — without them the alternation binds to the whole pattern
instead of the tail. - Collapse the heaviest class to a bare
Cls::if it still does not fit, largest
first, stopping as soon as it fits. That is a strict superset of the original selection
(2894 vs 2243 tests on the measured case), so it can only ever run MORE tests — it can
never turn a killed mutant into a survivor. - Budget below the hard cap (we use 96 KiB): a filter sized to the exact limit is one new
test away from failing again. - Throw with the measured length if even a full collapse does not fit, rather than
dropping the filter. Dropping it would run the whole suite per mutant and look like a
fast green. - Print whenever a collapse fires — which classes, how many tests it widened, the
resulting length. A silent widening would make the score certify more than the run
measured.
One caveat worth documenting alongside it: widening is safe only while the ordinary
suite is green. A test already failing for unrelated reasons counts as a kill.
Result on the case above: 163,246 → 95,614 bytes, four classes collapsed, no test lost.
We run this as a composer-patches patch today and would happily send it as a PR if the
approach looks right. We did not want to open with an unsolicited patch to a file that
already carries a TODO about redesigning this exact thing.
Environment
pestphp/pest-plugin-mutatev4.0.1,pestphp/pest4.7.5, PHP 8.4- Linux amd64 container (fails), macOS arm64 (does not — no per-element cap)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pest-plugin-mutate/src/MutationTest.php at MutationTest::start() and the filter-building loop around line 92; reproduce the Linux failure with the documented widely-covered class. Implement and validate the agreed bounded encoding, widening diagnostics, and measured-length failure, then confirm the resulting filter stays below the budget and remains compilable by PCRE.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, php
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100