BOHICA-LABS / BOHICA-LABS/vsdd-factory
Add a hollow-demo / false-confidence checker (agent + skill + gate integration)
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
Add a first-class **hollow-demo / false-confidence checker** to vsdd-factory — an agent + skill that detects artifacts which *execute and pass* but do not actually establish the behavior they claim. The adversary catches defects **within** the perimeter it's shown; hollow demos survive because the perimeter itself (the test or demo) is vacuous.
## Motivation (real-world evidence)
On a live VSDD project (algorithm-first trading-strategy validation), a fresh-context, codebase-wide sweep found **3 CRITICAL + 14 IMPORTANT** hollow-demo issues across **already-merged** code that had each passed per-story adversarial convergence (3+ clean passes) and wave integration gates. Representative findings:
- The product's central decision (a GO/NO-GO verdict) was **never exercised end-to-end on real computation** — every real run degenerated to `INCONCLUSIVE`, and *all* GO/NO_GO assertions fed hand-built fixtures to a pure-function seam, never the actual pipeline.
- A gate-critical metric (Monte-Carlo p95 drawdown) had **no independent value oracle** — only type/bounds/direction/determinism checks. A p90-vs-p95 or scaling bug would ship green.
- Demo evidence **documented a since-removed bug as the live mechanism**, inviting its re-introduction.
- Silent fallbacks (`None`→`0.0`, fabricated confidence intervals, swallowed `except Exception`, `getattr(...,"converged",True)`) masked unmet behavior with no degraded flag.
- Zero-assertion "characterization/informational" `test_*` functions inflated the green count (pass against a no-op implementation).
None of these are within-diff defects an adversary reviewing a PR would reliably catch. They require a dedicated lens that asks, for every test/demo: **"what wrong behavior would this FAIL to catch?"**
## Proposed capability
**1. Agent `hollow-demo-checker`** (read-only: Read/Grep/Glob/Bash) with this taxonomy:
- Vacuous tests — assert only "no exception"/existence/tautology; would pass against a stubbed/constant/degenerate impl.
- Silent fallbacks masking unmet behavior — swallowed exceptions; None/empty/NaN coerced to a "safe" default; skip-the-substance branches.
- Fail-open gate-critical metrics — a metric that PASSES its gate when it could not be computed.
- Fabricated/placeholder data substituting for real computation — hand-built objects / synthetic matrices standing in for real results.
- Demos that don't exercise the real path — canned output; `[PASS]` not backed by a real assertion; run only on trivial/synthetic data that cannot reveal a wrong answer.
- Lookahead / leakage presented as valid.
- Claimed-but-undemonstrated AC/BC clauses.
- **Disclosure discriminator:** separate DISCLOSED-acceptable (documented limitation / tech-debt / structured warning) from UNDISCLOSED (finding).
**2. Skill `/hollow-demo-sweep`** — fan-out read-only auditors per subsystem, synthesize, emit a findings register (severity + disclosure + remediation grouping).
**3. Integration points:**
- per-story-delivery Step 4.5 — add a hollow-demo lens to adversarial convergence (every AC needs a *substance* assertion, not just "executes").
- wave-gate, Phase 5 (adversarial refinement), Phase 7 (convergence).
- maintenance-sweep — periodic codebase-wide audit.
- optional lint-style hook — AST scan for assertion-free `test_*`, bare `except`, None-coercion of gate metrics.
## Acceptance criteria
- [ ] `agents/hollow-demo-checker.md`
- [ ] `skills/hollow-demo-sweep/SKILL.md` + taxonomy reference doc
- [ ] wired into per-story-delivery, wave-gate, phase-5, maintenance workflows
- [ ] findings-register template
_Reported from a real drift-lab cycle: a codebase-wide sweep found 3 CRITICAL + 14 IMPORTANT hollow-demo issues post-merge, all having previously passed adversarial convergence._
Contributor guide
Assessment
This issue has not been assessed yet.