BOHICA-LABS / BOHICA-LABS/vsdd-factory
adversary/test-writer policy: a "reference oracle" that duplicates the production mapping it validates gives zero drift protection (false independence claim)
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
A verification-property / behavioral-contract test frequently guards a production mapping (e.g. state → display-state, input → classification) by comparing production output against a **"reference oracle"** defined in the test file. In an observed case, the reference oracle was a **byte-for-byte duplicate of the production mapping function** — identical match arms, identical string literals — with a test-file doc comment claiming it was "an independently-derived oracle."
Because the proptest calls its own copy (not the production function), and the copy is identical, the oracle provides **zero drift protection**: if production drifts (e.g. a match arm is changed by mistake), the oracle must be hand-edited to match, and until it is, the proptest still passes against the *old* copied logic while never invoking production at all. The two "sides" of the equivalence check are the same code, so they can only ever agree. The test's stated purpose (catch production-mapping drift) is unmet, and the doc comment asserting independence is false.
This is adjacent to but distinct from #364 (test *name/comment* claims coverage the assertion doesn't exercise). Here the assertion *does* run and *does* pass — it's the **oracle's independence claim** that's false, and the structural problem is duplication, not a mislabeled assertion.
## Why it matters
Reference-oracle tests are a standard and valuable pattern (differential testing), but ONLY when the oracle is genuinely independent — a simpler/alternate formulation, a table of hand-written expected pairs, or the production function itself called directly with an independent expectation set. A copied oracle is strictly worse than no oracle: it costs maintenance, gives false confidence, and its "independent" doc comment misleads reviewers into thinking drift is covered.
## Expected behavior / suggested policy
- `test-writer` (and an `adversary` policy check) should flag a test-defined oracle that is structurally identical to the production function it validates.
- Acceptable forms:
1. **Call production directly**, assert against an independently-authored expectation set (hand-written pairs, or a property that doesn't restate the mapping).
2. **Alternate formulation** oracle (different structure that would diverge under the same bug).
3. If the production mapping is exhaustively small, a **pinned table unit test** (`f(A) == expected_A`, ...) authored from the spec, not copied from the impl.
- Any oracle whose doc claims "independent" must not be a copy — this is a checkable invariant (compare oracle body against the cited production function).
- Make the production mapping test-accessible (`pub(crate)` + test-helper) so option 1 is available rather than forcing a copy.
## Suggested area
`test-writer` oracle-construction guidance; `adversary` policy to detect duplicated oracles claiming independence. Cross-ref #363, #364.
Contributor guide
Assessment
This issue has not been assessed yet.