standards: assert the outcome not the call, and mutation-verify bug-fix tests

Problem

Two failures in one session on TimZander/AudioClassifier #195, same root cause: a test asserted the mechanism instead of observing the behaviour, and stayed green while the code was wrong.

Case 1 — the test that certified a regression

A fix bounded a service-worker fetch with AbortSignal.timeout(5000). The test asserted AbortSignal.timeout was called with 5000. It passed.

The bound was a regression. fetch() resolves at the headers, so the abort tore the response body afterwards — after .then had already handed the response to the client — meaning the .catch(() => caches.match(request)) fallback never ran. A slow load became a broken one. Measured in Chromium:

headersResolvedAt:        fetch() promise RESOLVED (headers received)
catchFiredOnFetchPromise: false                     <-- fallback never runs
bodyRead:                 THREW: AbortError
control (no signal):      status=200 body=31 bytes  <-- completes fine

The stub AbortSignal.timeout never aborted anything and the fake response had no body stream, so the test was structurally incapable of telling a working bound from a harmful one. The defect survived a full round of review-and-fix behind a green suite.

Case 2 — the suite that pinned nothing

The same PR's first test file let 7 of 8 mutants survive, including deletion of the very cachePut call the PR's branch depended on, because the stubs recorded no writes. It looked like coverage. It was decoration.

Case 3 — the claim that outran the evidence

After rewriting, I reported "13 of 13 mutants caught". A later review found a mutant I had not thought of — deleting the type half of a guard — survives, and that a test claiming to pin that guard could never reach it (ok: false short-circuits first). The mutant set bounds the claim.

Proposed rules for standards/CLAUDE.md

Under Unit Test Standards → How to write unit tests:

Assert the outcome, not the call. If a test can only assert that a function was invoked with a particular argument — rather than observing what that argument caused — it cannot distinguish a working implementation from a broken one. Treat "I can only check the parameter" as a signal that the real check belongs in an integration or browser-driven test. Say so in the test file rather than letting a green run imply coverage.

Document a harness's blind spots inside the harness. When a test double is materially simpler than the real API (no body streams, no redirects, exact-match lookups instead of real matching semantics), state what it cannot observe at the top of the file, with issue links. A green run must never be read as excluding a bug the harness structurally cannot see.

Under Unit Test Standards → When to write unit tests (bug fixes):

Mutation-verify every bug-fix test. Revert the fix, run the test, confirm it fails, restore the fix. A test that passes both ways pins nothing, and "the test passes" is not evidence the fix works. When the suite is new, extend this to the code around the fix: mutate a neighbouring behaviour (delete a call, invert a condition, drop a guard) and confirm something fails. Surviving mutants are uncovered behaviour.

Report mutation results honestly: the mutant set you choose bounds what you can claim. "All mutants caught" only ever means "all the mutants I thought of".

Acceptance criteria

• "Assert the outcome, not the call" added under how-to-write
• "Document a harness's blind spots inside the harness" added
• "Mutation-verify every bug-fix test" added under bug-fix testing, with the revert → run → confirm-fail → restore loop
• The caveat that the mutant set bounds the claim is stated explicitly

Notes

The standards already say "Tests that pass regardless of whether the feature works are worthless" and "Is the test testing implementation or behavior?". This adds the missing operational half: how you find out, and the specific trap (asserting a call) that produces a worthless test while looking rigorous.

Possible follow-up tooling: a /mutation-check skill to run the loop mechanically — filed separately.

Open standards/CLAUDE.md and locate the Unit Test Standards sections named in the issue. Add the four proposed rules under the specified headings, preserving the existing guidance and explicitly including the revert → run → confirm-fail → restore loop and mutant-set caveat. Done means every acceptance criterion is checked.