BOHICA-LABS / BOHICA-LABS/vsdd-factory
enhancement(adversary): require scan-plan declaration and coverage report in adversarial pass output
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
The adversary agent's prompt declares a scan perimeter (e.g., `.factory/specs/` + `.factory/holdout-scenarios/`), but does not require the adversary to enumerate its scan plan at the start or produce a coverage report alongside findings. As a result, each fresh-context pass attacks a different region of the spec corpus — finding novel defects, yes, but with no auditable record of WHICH regions it attacked and which it skipped.
This has two consequences:
1. **Convergence non-monotonicity**: pass-7 spike (19 findings) vs pass-8 plateau (12) vs pass-9 (15) is partially driven by which regions each pass attacks. Without coverage records, the orchestrator cannot determine "did pass-9 actually re-cover the regions pass-8 attacked, or did it just look elsewhere?"
2. **Blind spots persist**: some files appear in zero findings across all passes — verification-properties/, certain UX shards, certain holdout scenario edge cases. Are these clean, or are they unread? Currently unknowable.
## Reproduction
ftc-blue Phase 1d cycle (2026-06-23):
- Pass-5 attacked: BC body NFR labels, NFR catalog citations
- Pass-6 attacked: character_unlock_state Dict vs Array, ER schema, hint-system residue
- Pass-7 attacked: FR-numbering namespace, EventBus catalog, save-schema nesting, matrix arithmetic (FOUR genuinely different regions)
- Pass-8 attacked: DI-NNN fork, emitter chains, dual-store changelog, signature arity
- Pass-9 attacked: era_id convention, matrix arithmetic (VP-014, different from pass-7), AC-NNN citation validity
No single pass had complete coverage of the corpus. No two passes attacked exactly overlapping regions. Coverage is implicit and ad-hoc.
## Proposed change
Update the adversary agent's prompt template to require:
1. **Pre-scan plan declaration**: At the start of the response, before findings, the adversary declares its scan plan. Example:
```
## Scan Plan
- Behavioral contracts: 74 files in .factory/specs/behavioral-contracts/
- Holdout scenarios: 50 files in .factory/holdout-scenarios/
- Architecture: 8 ADRs + 9 subsystem shards + 7 top-level + 3 verification docs
- Domain spec: capabilities (14) + entities (5) + invariants (3) + state-machines (5)
- UX: 10 screens + 4 interactions + 5 DS shards + 2 indexes
- PRD supplements: L3-PRD, NFR catalog, error taxonomy
- Index files: BC-INDEX, ARCH-INDEX, HOLDOUT-INDEX, UX-INDEX, EVAL-INDEX, L2-INDEX
Total in-scope: ~190 files. I will read each.
```
2. **Coverage report at the end**: After findings, summary of which files were sampled deeply, which were skimmed, which were skipped (with reasoning).
```
## Coverage Report
- Deep-read: 47 files (those that produced findings or were canonical sources)
- Skim-read: 102 files (cross-reference checks only)
- Unread: 41 files (out-of-scope or skipped due to budget; list below)
- Unread list:
```
3. **Pass-over-pass coverage tracking**: The orchestrator (or a coverage-meta skill) aggregates per-pass coverage reports into a cumulative matrix: which files have been deep-read in N passes; which have never been read. This surfaces blind spots.
## Applies to
- `agents/adversary.md` — prompt template
- New skill or workflow step: `phase-1d-coverage-aggregation` — consumes per-pass coverage reports
- `templates/adversary-report-template.md` — add Scan Plan + Coverage Report sections
## Acceptance criteria
- [ ] Adversary reports include Scan Plan and Coverage Report sections
- [ ] Orchestrator can produce a cumulative-coverage matrix across N passes
- [ ] Blind spots (files unread across N passes) surface automatically
- [ ] At least one cycle demonstrates better convergence trajectory due to coverage-aware adversarial dispatch
## Found during
ftc-blue Phase 1d adversarial cycle (2026-06-23, vsdd-factory@1.0.0-rc.21). 5 passes; no two attacked overlapping regions; cumulative coverage status was unknown to the orchestrator.
## Notes
This addresses a structural reason convergence is hard: without coverage discipline, fresh-context passes are forever finding "new" regions of the same finite spec. Making coverage explicit lets the orchestrator drive the adversary toward saturation.
Contributor guide
Assessment
This issue has not been assessed yet.