New skill: mutation-test a change set to find tests that pass against broken code

Goal

A skill that mutation-tests a change set: break the source in targeted ways, run the suite, and report which mutations survived — because a surviving mutation is a test that would pass against broken code.

Why a skill rather than documentation

Three things make this a poor fit for a written rule and a good fit for automation.

It pays out every time. In a single session it was run four times — twice by hand, twice by review agents — and found real defects on every run:

Two of those tests had docstrings asserting precisely the behaviour they failed to pin. Reading them did not reveal it; mutating did.

It has a destructive step with a known footgun. The restore must be done by copying back a backup taken beforehand — never git checkout --, which silently reverts uncommitted work in the same file and leaves the run looking green. That is already recorded as a hard-won lesson, and a written rule is exactly the kind of thing that gets skipped at step 9 of 12.

It is a fixed loop with a mandatory verification tail. Back up → mutate → run → record → restore → prove the tree is byte-identical. The last step is the one people drop, and dropping it is how a mutation ships.

Proposed behaviour

Given a target (changed files vs a base branch, or an explicit list) and a test command:

1. Refuse to start on a dirty tree, or require an explicit acknowledgement — the whole procedure assumes a known-good baseline.
2. Take a file-copy backup outside the repo.
3. Establish the baseline: the suite must be green before any mutation, or stop.
4. Apply each mutation, run the suite, record caught/survived, restore from the backup copy before the next one.
5. On completion, assert the working tree is byte-identical to the backup (diff per file, plus git status --porcelain unchanged) and say so explicitly in the output.
6. Report a table of mutation → caught/survived, with survivors called out as the finding — plus the exact command to reproduce each survivor.

Mutation selection should be driven by the diff's semantics, not a generic operator set: invert a new guard, drop a new argument at one call site, revert a changed default, make a conditional constant, remove a newly-added filter. The value came from mutations chosen to match what the change claims to do — a blanket "flip every > to >=" pass would have found none of the four results above.

Acceptance criteria

• Refuses to run against a dirty working tree without explicit acknowledgement.
• Verifies the suite is green before mutating and aborts if not.
• Restores by copying from a backup; never invokes git checkout, git restore, or git stash to undo a mutation.
• Verifies and reports that the tree is byte-identical when finished — a run that cannot prove this is a failed run.
• Reports survivors distinctly from caught mutations, with a reproduction command for each.
• Handles a mutation that makes the suite hang or crash (timeout, then restore) rather than leaving the tree mutated.
• Distinguishes a genuine catch from a flaky failure — re-run a "caught" result once when the suite is known-flaky, since a flake reads as a catch and hides a survivor. (This happened: two intermittent timeouts produced false catches in one campaign.)
• Works with an arbitrary test command, not one ecosystem's runner.

Open questions

• Should it propose the mutation list itself from the diff, take one from the caller, or both? The best results came from a caller-supplied list grounded in the change's claims.
• Is a survivor always a defect? Sometimes it means the behaviour is genuinely untestable at that layer and the honest output is a documented gap rather than a new test.
• Should it offer to write the missing tests, or stop at the report? Stopping is more trustworthy; the survivors are the deliverable.