BOHICA-LABS / BOHICA-LABS/vsdd-factory

policy(test-writer+adversary): panic-recovery negative tests must assert on the panic message, not just recover() != nil

Open
#402 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

Go / Rust negative tests that assert "input X causes a panic" frequently use only a bare recover-and-check-non-nil:

```go
defer func() {
if r := recover(); r == nil {
t.Fatal("expected panic")
}
}()
NewPathTrackerWithAddr(invalidAlpha, addr)
```

This is vacuous: it passes on ANY panic — nil-deref, wrong-field-check, unrelated bug — not just the panic the test claims to guard. The named oracle (\"invalid alpha rejected\") is not enforced by the assertion.

## Concrete evidence (switchboard-blue Wave-6 Tranche B)

`TestBC_2_06_003_NewPathTrackerWithAddr_RejectsInvalidAlpha` (S-BL.ROUTER-ADDR) survived multiple passes with `recover() == nil` as its only oracle. Pass-4 L2 hardened it to also check that the recovered panic message contains \"alpha\" — commit d04bf86.

This is the panic-message analog of two existing issues:
- #360 (panic-sourced RED tests → vacuous-GREEN after impl removes the panic)
- #364 (test name claiming branch coverage the assertion doesn't exercise)

But it's a distinct mechanism: here the test is still \"assertion-sourced\" (the recover check exists), but the assertion is polarity-only, not identity-verifying. Any panic satisfies it.

## Proposed remediation

**test-writer prompt rule:** panic-recovery negative tests MUST assert on the recovered value's message content — grep for the domain term the test name promises. Boilerplate:

```go
defer func() {
r := recover()
if r == nil {
t.Fatal(\"expected panic\")
}
msg := fmt.Sprint(r)
if !strings.Contains(strings.ToLower(msg), \"alpha\") {
t.Fatalf(\"expected panic mentioning 'alpha', got: %s\", msg)
}
}()
```

**adversary Ln policy:** flag any panic-recovery test whose only oracle is `r == nil` when the test name promises branch-specific coverage — MEDIUM severity, since polarity-only assertions fail the same discipline as #364.

**deterministic lint (per #336):** grep for `recover()` in `_test.go` where the enclosing function's name contains \"Rejects\" / \"Invalid\" / \"Panics\" and the block lacks a `Contains` / `String()` inspection.

## Cross-references

- #360 — vacuous-green panic-sourced red tests
- #364 — test name/comment vs assertion branch mismatch (same discipline, different fingerprint)
- #336 — deterministic pre-review lint layer

## Severity

MEDIUM. Recurred through 3+ passes on switchboard-blue Wave-6 Tranche B; not caught by any current gate.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.