BOHICA-LABS / BOHICA-LABS/vsdd-factory
policy(test-writer+adversary): panic-recovery negative tests must assert on the panic message, not just recover() != nil
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
Go / Rust negative tests that assert "input X causes a panic" frequently use only a bare recover-and-check-non-nil:
```go
defer func() {
if r := recover(); r == nil {
t.Fatal("expected panic")
}
}()
NewPathTrackerWithAddr(invalidAlpha, addr)
```
This is vacuous: it passes on ANY panic — nil-deref, wrong-field-check, unrelated bug — not just the panic the test claims to guard. The named oracle (\"invalid alpha rejected\") is not enforced by the assertion.
## Concrete evidence (switchboard-blue Wave-6 Tranche B)
`TestBC_2_06_003_NewPathTrackerWithAddr_RejectsInvalidAlpha` (S-BL.ROUTER-ADDR) survived multiple passes with `recover() == nil` as its only oracle. Pass-4 L2 hardened it to also check that the recovered panic message contains \"alpha\" — commit d04bf86.
This is the panic-message analog of two existing issues:
- #360 (panic-sourced RED tests → vacuous-GREEN after impl removes the panic)
- #364 (test name claiming branch coverage the assertion doesn't exercise)
But it's a distinct mechanism: here the test is still \"assertion-sourced\" (the recover check exists), but the assertion is polarity-only, not identity-verifying. Any panic satisfies it.
## Proposed remediation
**test-writer prompt rule:** panic-recovery negative tests MUST assert on the recovered value's message content — grep for the domain term the test name promises. Boilerplate:
```go
defer func() {
r := recover()
if r == nil {
t.Fatal(\"expected panic\")
}
msg := fmt.Sprint(r)
if !strings.Contains(strings.ToLower(msg), \"alpha\") {
t.Fatalf(\"expected panic mentioning 'alpha', got: %s\", msg)
}
}()
```
**adversary Ln policy:** flag any panic-recovery test whose only oracle is `r == nil` when the test name promises branch-specific coverage — MEDIUM severity, since polarity-only assertions fail the same discipline as #364.
**deterministic lint (per #336):** grep for `recover()` in `_test.go` where the enclosing function's name contains \"Rejects\" / \"Invalid\" / \"Panics\" and the block lacks a `Contains` / `String()` inspection.
## Cross-references
- #360 — vacuous-green panic-sourced red tests
- #364 — test name/comment vs assertion branch mismatch (same discipline, different fingerprint)
- #336 — deterministic pre-review lint layer
## Severity
MEDIUM. Recurred through 3+ passes on switchboard-blue Wave-6 Tranche B; not caught by any current gate.
Contributor guide
Assessment
This issue has not been assessed yet.