BOHICA-LABS / BOHICA-LABS/vsdd-factory
process-gap(mutation-review): tautological zero-assertion tests are invisible to mutation analysis — mutation pass rates suites CLEAN while spec-conformance crosswalk finds unimplemented branches
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
Tests of the form `assert_true(true)` (\"phase invocable\" smoke tests, or any test with zero behavioral assertions) are structurally invisible to mutation analysis: they kill no mutants and survive all of them, so a mutation analysis pass rates the suite CLEAN while a spec-conformance clause-crosswalk pass on the same story finds entire mandated branches unimplemented behind them. The two lenses (mutation analysis vs. conformance crosswalk) are not redundant — they test orthogonal properties — but a mutation pass that does not first exclude tautological tests produces a false-confidence CLEAN verdict.
## Observed pattern (private Godot 4.7 project; identifiers scrubbed)
A story's test suite included \"phase invocable\" smoke tests: tests that assert the subsystem can be instantiated and a method can be called without crashing. No behavioral assertions on output. The mutation analysis pass rated the suite CLEAN: no surviving non-equivalent mutants detected. A separate spec-conformance crosswalk pass found that entire mandated behavioral branches (error path, state-transition invariant) were unimplemented — the corresponding test stubs existed but contained only `assert_true(true)`.
The mutation analysis was technically correct: the suite was mutation-clean *for the assertions it made*. But the assertions made were vacuous — mutation analysis cannot distinguish between \"the test verifies behavior and kills mutants\" and \"the test makes no claims so there are no mutants to kill.\"
This recurrence directly supports the data point in #462 where the two-lens composition was proposed: mutation analysis validates tests that ASSERT; only spec-conformance crosswalk validates that tests assert THE RIGHT THING.
## Why this matters
- Mutation analysis is the strongest quantitative evidence for test suite quality, so a CLEAN verdict carries high reviewer confidence.
- A CLEAN verdict on a suite with tautological tests is structurally false: it does not mean the suite validates behavior; it means the suite does not contradict any mutant.
- Reviewer briefs that present mutation analysis without flagging zero-assertion tests pass false confidence through the convergence gate.
- The defect class is invisible to any lens that relies on the mutation pass verdict without examining assertion density.
## Concrete recurrence datum (support for #462 lens-rotation recommendation)
This is a recurrence of the lens-rotation finding documented in #462: the two lenses are not redundant. In the #462 comment on this issue, the finding was framed as a methodology recommendation. This instance provides a concrete datum: a mutation pass produced a CLEAN verdict on a suite where conformance crosswalk found entire mandated branches unimplemented. The mutation pass was not wrong; the reviewer brief was wrong for treating CLEAN mutation as sufficient evidence.
## Proposed remedy
**Reviewer briefs for mutation analysis passes should first inventory zero-assertion and tautological tests and exclude them from coverage claims before running the mutation analysis:**
1. Pre-mutation audit: identify tests with zero assertions (or only `assert_true(true)` / `assert_false(false)` pattern) — these are tautological and produce no mutation signal.
2. Report tautological test count and which ACs/ECs they claim to cover. If the count is nonzero, the mutation CLEAN verdict does not apply to those ACs/ECs.
3. Require conformance crosswalk on any AC/EC covered only by tautological tests — mutation analysis cannot substitute.
4. Alternatively: gate mutation analysis on a minimum assertion-density check per test (e.g., every test must have at least one non-tautological assertion before it is eligible to kill a mutant).
This complements the coverage-manifest gate proposed in #353 (per-AC/per-EC behavioral vs structural tagging): a coverage manifest that distinguishes \"behavioral-assertion\" from \"invocation-only\" tests would expose the tautological class before mutation analysis is run.
## Related
- #462 (probe variance / lens rotation) — the two-lens composition (mutation analysis + spec-conformance crosswalk) is proposed there as a methodology mandate; this issue provides a concrete recurrence datum
- #353 (coverage-manifest gate, per-AC/per-EC behavioral|structural tagging) — tautological tests would be tagged \"structural\" (invocation-only) and their ACs flagged for crosswalk coverage
- #475 (stub-architect test-file authorship) — some tautological tests originate as stub-stage test scaffolding that never gets behavioral assertions added; the two issues share a root (early-stage test artifacts with vacuous assertions surviving into convergence)
Contributor guide
Assessment
This issue has not been assessed yet.