BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(test-writer+orchestrator): high vacuous-pass ratio at RED gate dispatch is unenforced — stubs incidentally satisfy behavioral assertions

Open
#353 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

**Pattern: high vacuous-pass ratio at RED gate dispatch goes undetected**

In this session's parallel Phase-3 dispatch (4 stories, fresh test-writer agents), one story shipped its RED suite with **16 of 34 new tests already green** before any implementation. The test-writer agent flagged the smell in its report; the orchestrator and downstream gates did not catch it as a gating-blocker.

### Why it happens

Stub-architect produces minimum-viable stubs (`push_error("STUB: …")` bodies, default-zero fields). Test-writer writes assertions against those stubs. Two failure modes:

1. **Field-default tautology**: a test asserts `audio_manager.corruption_volume_linear == 0.0` against a default-initialized float that hasn't been touched — passes on RED, passes after impl writes the same value differently, never fails on a wrong impl that leaves it at zero.
2. **Method-existence guard**: a test asserts `has_method("foo")` against a stub method that was registered for compilation — passes on RED, passes forever, asserts nothing behavioral.

GUT's RED gate is currently "the test FILE runs without parse error and at least some tests fail." A 16/34 vacuous-pass ratio meets that bar but defeats the spirit of TDD discipline.

### Signal that should be enforced

When a test-writer reports `red_count / total_count < ~0.7` for a freshly-introduced BC test file, the orchestrator should require a remediation pass before accepting the RED gate. The remediation:

- Strengthen the tests (add behavioral assertions that fail on default state).
- Or downgrade the marked-tests-on-RED to a separate file labeled `test_invariants_*.gd` so they're not counted as BC-coverage.
- Or accept and explicitly mark them GREEN-BY-DESIGN in the test file with a structured comment, which the adversary then audits.

### Why it survives current gates

- BC-5.39.001 adversarial convergence kicks in **after** GREEN, by which time the implementer has already invested ~30-60 min on a foundation with weak negative assertions.
- The adversary catches tautologies if it explicitly probes them, but its perimeter is "implementation vs BC," not "test-suite quality given current stub state."
- No `[red_quality]` finding category in the adversary tags.

### Suggested intervention

Either:
(a) test-writer agent prompt mandates `red_ratio >= 0.7` and a self-strengthening loop before dispatching, or
(b) a deterministic pre-adversary lint pass (#336 style) that diffs the pre-impl test outcomes against post-impl and warns when the delta is anomalously small.

### Provenance

Observed 2026-06-29 in a 4-way parallel TDD dispatch. The implementer for the affected story was instructed (after the smell was noted in human ledger) to strengthen 10 of the 16 tests; 6 remained as legitimate seam-guards. The fix worked but required out-of-band intervention.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.