[Feature Request]: `--update-mutation-shards`: time-balanced sharding for `--mutate --shard`
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 11.7k
- Forks
- 538
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 8
Description
Problem
--update-shards is silently ignored when combined with --mutate. Running:
./vendor/bin/pest --mutate --parallel --update-shards
runs the tests and mutations correctly, but shards.json is never written. The EnsureShardTimingStarted/EnsureShardTimingFinished PHPUnit subscribers that collect test class timings don't produce meaningful data during a mutation run — mutation workers re-run filtered test subsets against mutated source files, not full test classes — so the timings collection returns empty and the file write is skipped silently.
As a result, when using --mutate --shard=N/T, the Shard plugin has no mutation-aware timings to work with. It either falls back to shards.json (test execution timings, which are unrelated to mutation cost) or partitions test classes by count. Either way, shards are badly unbalanced in practice because mutation time is not correlated with test runtime — a test class running in 0.06s may cover a large source file generating 200 mutations, while a test taking 10s may cover almost nothing mutable.
Real-world example from CI with 4 shards:
| Shard | Time |
|---|---|
| 1 | 8m 05s |
| 2 | 8m 53s |
| 3 | 3m 02s |
| 4 | 3m 20s |
The total wall time is bottlenecked by shard 2 at nearly 9 minutes, even though shards 3 and 4 are mostly idle.
Proposed solution
Introduce --update-mutation-shards, analogous to --update-shards, which runs the full mutation suite, records the total mutation execution time per test class (by summing MutationTest::duration() across each MutationTestCollection), and writes tests/.pest/mutation-shards.json in the same format as shards.json.
When --mutate --shard=N/T is active and mutation-shards.json exists, Shard.php would load it in preference to shards.json for the LPT partitioning step.
Suggested workflow
# weekly CI job — generates the timing file
./vendor/bin/pest --mutate --parallel --covered-only --update-mutation-shards
# regular CI shard jobs — now time-balanced by mutation cost
./vendor/bin/pest --mutate --min=100 --parallel --covered-only --shard=${{ matrix.shard }}/4
Implementation sketch
The data is already available:
MutationRepository::all()returnsMutationTestCollection[]keyed by source file pathMutationTest::duration()tracks per-mutation execution time
The main design question is where the grouping lives — pest-plugin-mutate has the mutation timing data, while Shard.php in pestphp/pest owns the partitioning and file writing. This likely needs a small touch in both repos, or a hook that lets pest-plugin-mutate supply an alternative timings source to Shard.php.
Would you welcome a PR for this? Happy to take a first pass if the approach sounds right, or adjust the design based on your feedback.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading Shard.php in pestphp/pest and the mutation timing sources MutationRepository::all() and MutationTest::duration() in pest-plugin-mutate. Trace how shards.json is written and how mutation runs select shard timings. Done means --update-mutation-shards writes tests/.pest/mutation-shards.json and --mutate --shard uses it for LPT partitioning when present, with coverage for the new workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- cli, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100