BOHICA-LABS / BOHICA-LABS/vsdd-factory

adversary/test-writer policy: a "reference oracle" that duplicates the production mapping it validates gives zero drift protection (false independence claim)

Open
#381 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

A verification-property / behavioral-contract test frequently guards a production mapping (e.g. state → display-state, input → classification) by comparing production output against a **"reference oracle"** defined in the test file. In an observed case, the reference oracle was a **byte-for-byte duplicate of the production mapping function** — identical match arms, identical string literals — with a test-file doc comment claiming it was "an independently-derived oracle."

Because the proptest calls its own copy (not the production function), and the copy is identical, the oracle provides **zero drift protection**: if production drifts (e.g. a match arm is changed by mistake), the oracle must be hand-edited to match, and until it is, the proptest still passes against the *old* copied logic while never invoking production at all. The two "sides" of the equivalence check are the same code, so they can only ever agree. The test's stated purpose (catch production-mapping drift) is unmet, and the doc comment asserting independence is false.

This is adjacent to but distinct from #364 (test *name/comment* claims coverage the assertion doesn't exercise). Here the assertion *does* run and *does* pass — it's the **oracle's independence claim** that's false, and the structural problem is duplication, not a mislabeled assertion.

## Why it matters

Reference-oracle tests are a standard and valuable pattern (differential testing), but ONLY when the oracle is genuinely independent — a simpler/alternate formulation, a table of hand-written expected pairs, or the production function itself called directly with an independent expectation set. A copied oracle is strictly worse than no oracle: it costs maintenance, gives false confidence, and its "independent" doc comment misleads reviewers into thinking drift is covered.

## Expected behavior / suggested policy

- `test-writer` (and an `adversary` policy check) should flag a test-defined oracle that is structurally identical to the production function it validates.
- Acceptable forms:
1. **Call production directly**, assert against an independently-authored expectation set (hand-written pairs, or a property that doesn't restate the mapping).
2. **Alternate formulation** oracle (different structure that would diverge under the same bug).
3. If the production mapping is exhaustively small, a **pinned table unit test** (`f(A) == expected_A`, ...) authored from the spec, not copied from the impl.
- Any oracle whose doc claims "independent" must not be a copy — this is a checkable invariant (compare oracle body against the cited production function).
- Make the production mapping test-accessible (`pub(crate)` + test-helper) so option 1 is available rather than forcing a copy.

## Suggested area

`test-writer` oracle-construction guidance; `adversary` policy to detect duplicated oracles claiming independence. Cross-ref #363, #364.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.