BOHICA-LABS / BOHICA-LABS/vsdd-factory
process-gap(test-writer+stub-architect): panic-sourced red tests degrade to vacuous-green no-ops at Red→Green transition (green-side analogue of #353)
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
A TDD test whose Red-Gate "redness" is produced by the stub's `todo!()`/`panic!()` **panic** — rather than by a failing **assertion** — silently degrades into a trivially-green **no-op** the instant the implementation removes the panic. Nothing in the test-writer prompt, the stub-architect Red Gate, or the per-story Step 4→5 (Red→Green) transition detects or forbids this.
This is the **green-transition analogue** of #353 (which covers stubs *incidentally satisfying* a real assertion at red-dispatch time). Here there is **no assertion at all** — the test only ever "worked" because calling the stub panicked.
## Mechanism
Red Gate (BC-5.38.001) requires stubs to compile but fail. A test author can satisfy "fails while stubbed" two ways:
1. **Assertion-sourced red (correct):** call the unit, capture the result, `assert!` on observable behavior. Fails red because the stub returns the wrong/absent value; goes green when behavior is correct.
2. **Panic-sourced red (defective):** call the unit and discard the result (`let _ = unit.do_thing().await;`) with **zero assertions**. Fails red *only* because `todo!()` panics. The moment the body is implemented and returns normally, the test passes while asserting nothing — it would pass equally against a no-op implementation.
Type-2 tests look identical to type-1 at the Red Gate (both are "failing tests that compile"). They diverge only *after* Green, where they become permanent vacuous passes. By then the per-story convergence has moved on.
Observed concrete shape (generalized, names changed):
```rust
#[tokio::test]
async fn test_..._is_not_yet_implemented() {
let writer = FileWriter::new("/tmp/fixed-path.ndjson"); // also: fixed path, not a tempdir
let _ = writer.append(&entry).await; // result discarded, nothing asserted
// (relied on todo!() panic for redness; now a no-op)
}
```
These survived test-authoring, the Red Gate, and the Green transition; they were caught only by a later fresh-context adversarial pass (and only because a human directed scrutiny at test *quality*, not just the test *tally*).
## Why the existing gates miss it
- **stub-architect / Red Gate (#357, BC-5.38.001):** verifies tests fail while stubbed — a panic satisfies this perfectly.
- **#353 (vacuous-pass at RED dispatch):** scoped to stubs *incidentally satisfying assertions* — does not fire when there is no assertion.
- **adversary at per-story convergence (#298):** read-only, trusts the test tally; cannot distinguish a no-op pass from a real pass without reading every test body.
- **CI:** a no-op test that returns `Ok` is green.
## Proposed fix (deterministic, cheap — complements #336)
Add a deterministic pre-review lint (the #336 "mandatory deterministic lint layer") rule:
> **A `#[test]`/`#[tokio::test]` that invokes a production function and discards its result (`let _ = …`, or a bare expression-statement of a `Result`/`#[must_use]` type) while containing zero `assert!`/`assert_eq!`/`assert_ne!`/`assert_matches!`/`panic!`/`?`-propagating assertions is a vacuous test → FAIL the gate.**
Plus a test-writer prompt rule and a Green-transition checklist item:
> When the implementer removes a `todo!()`, every test that previously covered that unit must be re-checked: its redness must now be **assertion-sourced**, not panic-sourced. Tests named `*_not_yet_implemented` / `*_panics_until_implemented` must be converted to behavior assertions or deleted before the story leaves Red→Green.
Bonus (separate but adjacent, also observed): three of the no-op tests wrote to **fixed `/tmp` paths** rather than `tempfile::tempdir()` — cross-run residue + parallel-run collision risk. A lint that flags fixed `/tmp`/hardcoded absolute paths in tests would catch this class too.
## Severity
P2 process-gap. Not a halt/panic defect, but it silently erodes the core TDD guarantee: a green suite that proves nothing about the units these tests nominally cover. Detectable deterministically, so the fix is high-leverage.
## Cross-references
- #353 — red-dispatch vacuous-pass (assertion incidentally satisfied); this issue is the green-transition / no-assertion analogue
- #357 — Red Gate tests cannot land on main (panic-as-red friction is the same root concept)
- #336 — mandatory deterministic pre-review lint layer (natural home for the detector)
- #298 — read-only adversary trusts an unverifiable test tally (why this slips convergence)
- #332, #335 — other false-green / unasserted-clause families
Contributor guide
Assessment
This issue has not been assessed yet.