BOHICA-LABS / BOHICA-LABS/vsdd-factory
enhancement(orchestrator+adversary): mandatory deterministic pre-review lint layer before LLM convergence passes
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
In one external pilot run (~63 review pass-pairs across two phases, 0 lines of product code), the convergence loop repeatedly reset on defect classes that are mechanically enumerable — meaning a deterministic grep, schema check, or structural comparison would have caught every instance in one shot. Instead, each instance was surfaced one-per-pass by a fresh-context LLM reviewer, triggering a streak reset on a material-but-trivial-to-fix finding.
This enhancement proposes a **mandatory deterministic pre-review lint layer** that must reach GREEN before any LLM adversarial or consistency-validator pass is dispatched. The layer drains mechanically-enumerable defect classes wholesale, so that LLM pass budget is spent exclusively on semantic and behavioral defects that only cross-story reasoning can find.
---
## Evidence: enumerable classes drained in the pilot
| Class | How it was discovered | Instances eliminated in one burst |
|-------|----------------------|----------------------------------|
| TBD story anchors on active contracts post-decomposition | Manual grep after ~13 passes had surfaced 1–2 per pass | 35 in one burst |
| Cross-story "previous story intelligence" prose restatement drift | Adversary surfaced in a pass; triggered targeted sweep | ~5 findings |
| Story canonical-frontmatter fields missing | Consistency-validator surfaced ~15 passes after decomposition | All story files in one burst |
| Verification-property template conformance | `validate-template-compliance` run proactively | All VP files + index in one burst |
| Canonical-surface rename residuals (retired identifier surviving in harness, story body, architecture prose) | Surfaced 6 separate times across consecutive passes, each in a file type NOT covered by the prior grep scope | full-scope grep eliminated the class entirely |
The 35-in-1 result is the clearest quantitative evidence: at the 1–2-per-pass discovery rate the loop was achieving before the manual sweep, that class alone would have consumed ~17–35 more LLM review pass-pairs before the corpus was clean — on anchors alone.
---
## What makes the loop non-terminating on these classes
The convergence gate definition (N consecutive clean passes over a stochastic full-corpus LLM pass; any material change resets) interacts badly with mechanically-enumerable classes:
1. Reviewer A finds instance K of a rename residual. The remediating burst fixes instance K but not sibling instance K+1 in a different file type (e.g., harness files were swept but architecture-rationale prose was not).
2. Reviewer B on the next pass finds instance K+1. Streak resets.
3. The fix for K+1 is applied but K+2 exists in yet another file type not covered.
4. Repeat.
This pattern was observed for the rename-residual class across 6 consecutive passes. Each individual fix was correct. The loop structure guaranteed recurrence because no single pass swept the entire corpus — each LLM reviewer has bounded context and tends to focus on the file type that showed the symptom.
---
## Minimum lint-coverage checklist
These checks are deterministic, require no LLM reasoning, and should all pass GREEN before the first adversarial pass of any phase:
1. **No TBD story anchors** — no active-owned contract may carry a TBD story-anchor field after story decomposition is complete.
2. **Canonical surface agreement (intra-document)** — every story's architecture-mapping table must agree with its own tasks/AC body on every enum variant set, struct field set, function signature, and method arity named in both sections.
3. **Canonical surface agreement (inter-document, bidirectional)** — for every type/function surface appearing in BOTH a story AC/task description AND an architecture document, the definition must agree bidirectionally. Both propagation directions must be checked.
4. **Index ↔ detail reconciliation** — every index file's count, version, and ownership cells must agree with the corresponding detail file's heading, frontmatter version, and ownership declaration.
5. **Cross-story prose-restatement agreement** — every "previous story intelligence" / "from story-N" prose block must be diffed against the canonical definitions in the module-decomposition and entity-definition documents. Any field/type/variant name present in the restatement but absent from the canonical definition (or vice versa) is a blocker.
6. **Dependency-table version/license agreement** — every library entry in any story's dependency table must match the canonical entry in the architecture's module-decomposition section.
7. **Canonical frontmatter conformance on ALL detail files** — run `validate-template-compliance` (the engine's own shipped skill) against every story, contract, and VP file before any pass is dispatched.
8. **Rename-propagation completeness (corpus-wide)** — after ANY canonical-surface rename or retirement, the residual grep must run across the ENTIRE corpus including architecture docs, rationale documents, assessment narratives, and VP prose blocks — not only harness/story/index files. This must execute as the closing step of any remediating burst before the pass can be declared clean.
9. **VP harness compilability** — for every VP proof-harness skeleton, mechanically check receiver form, arity, constructor existence, and enum-comparison form against the canonical signatures. A `cargo check`-style gate against a stub crate would catch this class mechanically.
---
## This unifies existing per-class requests
The following existing issues each address one defect class from the list above. This enhancement is the unifying proposal none of them states — a single mandatory pre-review layer that runs all checks at once, before any LLM pass is dispatched:
- #309 — TBD anchor / stale-reference class
- #299 — index ↔ detail reconciliation
- #287 — frontmatter conformance
- #216 — cross-document consistency
- #308 — per-finding CI job proliferation concern
---
## Response to the #308 tension
#308 raises a valid concern: if every finding that surfaces in a convergence pass spawns a new CI job, the CI job set becomes unbounded. This enhancement resolves that tension directly. The pre-review lint layer is:
- **Fixed and bounded** — the checklist above is finite and specified in advance, not grown by per-finding CI job creation.
- **Deterministic** — every check is a grep, schema validation, or structural comparison. No LLM reasoning. No false-negative variation across runs.
- **Runs once per cycle** — the lint gate runs once before the first LLM pass of each phase, not once per finding. It does not proliferate.
The per-finding CI job model (#308's concern) is orthogonal. The pre-review lint layer replaces the need for reactive per-class CI jobs by preventing the enumerable classes from ever reaching an LLM pass in the first place.
---
## Relationship to genuine semantic defects
This enhancement does NOT argue that the convergence loop is pure thrash. In the same pilot run, an early pass found two genuine build-blocking semantic defects (incompatible type-surface definitions across stories; a proof obligation with no valid owner) that survived ~9 prior structural passes. These require LLM-level cross-story semantic reasoning to find — no deterministic lint can catch a type-surface naming contradiction that spans story-ownership rules. The correct framing: a mandatory pre-review lint layer eliminates the cosmetic/mechanical thrash class entirely, freeing every LLM pass to focus on defects of exactly that semantic class.
Contributor guide
Assessment
This issue has not been assessed yet.