BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(formal-verifier+test-writer): checker validated by author-shared fixtures fails self-application — mandatory self-application smoke row

Open
#496 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Pattern

A guard/checker was validated by a scenario matrix of empirical test cases authored by the same person who wrote the guard. The matrix covered 100% of the cases the author imagined. The guard passed across N empirical scenario rows, and 3 consecutive CLEAN convergence cycles produced no new findings.

On the 4th cycle, a clause-audit lens — reading the guard's logic against its own repository's real bytes rather than against the handcrafted fixtures — found that the guard failed against its own codebase. Root cause: the author's test fixtures and the guard shared the same byte-shape assumption about the watched pattern (in this case, an ASCII variant of a Unicode character that appears in the real codebase). The fixtures used the ASCII form; the repository bytes used the Unicode variant. The guard matched only the ASCII form; the repository bytes were invisible to it.

## Metrics

- Defect survived 3 consecutive CLEAN adversarial convergence passes (6 total cycles)
- Found on the 4th cycle by a clause-audit lens that ran the guard against the actual repository
- Scenario matrix: 100% pass at time of all three CLEAN verdicts
- Fix: extend guard to cover both byte forms; re-verify against repository

## Why author-shared fixtures are insufficient

When one person writes both the fixtures and the guard, the fixtures encode that author's assumptions about the input space. Variants of the watched pattern that differ in encoding, normalization, casing, or whitespace that the author did not think of will not appear in the fixture set. The guard then passes all fixtures — correctly — while being blind to real-world inputs that use the unrepresented variant.

This is distinct from mutation testing (which verifies that assertions in tests are tight) and from adversarial fixture generation (which targets logic branches). The failure here is a fixture-input-space coverage gap: the fixtures represent what the author imagined the input looks like, not what the input actually looks like in the system being guarded.

## Proposed remedy

Any delivered checker or guard that processes patterns from real artifacts (files, config, source code) **must include a mandatory self-application row**: run the guard against its own PR's diff / its own repository's bytes, capture the result, and include it in the acceptance evidence. A checker that cannot cleanly process its own repository is not ready to ship.

If the self-application row would trivially pass (e.g., the guard pattern is never present in the checker's own codebase), the row should document that absence explicitly — it is still a meaningful evidence point.

## Framework ask

1. **Delivery contract for checkers/guards:** stub-architect or story-writer emits a mandatory acceptance criterion: "Self-application: run guard against this PR's own diff; record result." The AC is SATISFIED only by attached command output.
2. **Adversarial lens rotation item:** when converging on any guard/checker story, one pass must include a "run it against its own codebase" step and verify the result manually before accepting the CLEAN verdict.
3. **Fixture authorship note:** scenario matrices authored by the same person who wrote the guard require a second author or automated input-space enumeration (encoding variants, normalization forms, casing, whitespace) before the matrix can be considered exhaustive.

## Cross-refs

- #475 (stub-architect self-attestation — same author writes both stub and companion tests; tautology loop)
- #285 (self-reported verification contradicted by independent re-execution — same class, different stage)
- #462 (lens rotation — clause-audit lens caught what 3 convergence passes missed)

*(Framework-pattern detail only. From a private project running the factory.)*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.