standards: mutation-test the system under test, not the assertion

Problem

standards/CLAUDE.md already says tests must "verify observable behavior, not implementation details" and warns against tests that "pass regardless of whether the feature works." It does not say how to prove a given test isn't one of those — and the obvious method is subtly wrong.

The intuitive check is: change what the test expects, confirm it fails. That proves the assertion is wired up. It proves nothing about whether the test can detect the failure it exists to catch, because the input was never disturbed.

Evidence

Three vacuous tests shipped or nearly shipped in one session on TimZander/AudioClassifier, all with green suites:

1. A smoke test asserting deployed Cache-Control passed against a deploy with every file deleted. The edge derives the header from the URL, not the file — a deleted /model/BirdNET.tflite still answers max-age=31536000, immutable, because the rule matches the path and gets applied to the SPA fallback HTML. I "mutation-tested" it three ways (claim app.js is immutable → fail; claim vendor/ is no-cache → fail; expect a 404 → fail) and reported it verified. Every one of those mutated an expectation. None deleted a file. Caught by the reviewer, not by me.

2. A cross-file version test could not detect the bug it existed for. It grouped by source file, last-write-wins, so index.html's three references to the same icon collapsed to one. Bumping ref #1 or #2 passed 6/6; only ref #3 failed. The PR body claimed "bumping one index.html ref fails 1 test" — true for exactly one of three, evidently the one that was tried.

3. A unit test certified an offline path that fails in a real browser. Its stub modeled the cache as an exact string lookup with no URL list, so it could not see the redirect that breaks it. The bug was later reproduced in headless Chromium — the test asserting the guarantee was worse than no test, because it retired the question.

Common thread: each test's own anti-vacuity check mutated the expectation and stopped there.

Proposed — add to standards/CLAUDE.md under Unit Test Standards

Prove a test is not vacuous by breaking the system, not the assertion.

Changing what a test expects only proves the assertion is wired up. To show a test detects the failure it exists to catch, disturb the input: delete the file, corrupt the data, remove the config, revert the fix. Then confirm it fails.

• Mutate every instance, not a representative one. If a rule spans N call sites, break each independently — a check that collapses duplicates passes for some and fails for others, and "I tried one" finds the wrong one half the time.
• Ask what the test would do if the thing under test did not exist at all. If it still passes, it asserts something derived, not the thing.
• Distrust stubs on the failure path. A stub is a model of the dependency, and the bug usually lives in the difference. A test whose stub cannot represent the failure mode certifies the bug as fixed.
• Restore state after each mutation and confirm the tree is clean.

Why this is generic

Applies to any test in any language. The failure is a reasoning error about what a green assertion proves — orthogonal to framework, stack, or domain.

Acceptance criteria

• standards/CLAUDE.md's Unit Test Standards carries the rule.
• It states the distinction explicitly: mutating the expectation proves wiring; mutating the input proves detection.
• It includes the "every instance, not a representative one" clause.
• It includes the stub warning: a stub that cannot represent the failure mode certifies the bug as fixed.
• Cross-referenced from Code Review Standards ("Demand test coverage"), so a reviewer asks which mutation did you run rather than are there tests.

Start with standards/CLAUDE.md, reading the Unit Test Standards and the Code Review Standards section on “Demand test coverage.” Add the stated distinction between mutating expectations and inputs, plus the every-instance and stub warnings, then cross-reference the rule from Code Review Standards. Done means all listed acceptance criteria are reflected in that file.