BOHICA-LABS / BOHICA-LABS/vsdd-factory

enhancement(orchestrator+adversary): mandatory deterministic pre-review lint layer before LLM convergence passes

Open
#336 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

In one external pilot run (~63 review pass-pairs across two phases, 0 lines of product code), the convergence loop repeatedly reset on defect classes that are mechanically enumerable — meaning a deterministic grep, schema check, or structural comparison would have caught every instance in one shot. Instead, each instance was surfaced one-per-pass by a fresh-context LLM reviewer, triggering a streak reset on a material-but-trivial-to-fix finding.

This enhancement proposes a **mandatory deterministic pre-review lint layer** that must reach GREEN before any LLM adversarial or consistency-validator pass is dispatched. The layer drains mechanically-enumerable defect classes wholesale, so that LLM pass budget is spent exclusively on semantic and behavioral defects that only cross-story reasoning can find.

---

## Evidence: enumerable classes drained in the pilot

| Class | How it was discovered | Instances eliminated in one burst |
|-------|----------------------|----------------------------------|
| TBD story anchors on active contracts post-decomposition | Manual grep after ~13 passes had surfaced 1–2 per pass | 35 in one burst |
| Cross-story "previous story intelligence" prose restatement drift | Adversary surfaced in a pass; triggered targeted sweep | ~5 findings |
| Story canonical-frontmatter fields missing | Consistency-validator surfaced ~15 passes after decomposition | All story files in one burst |
| Verification-property template conformance | `validate-template-compliance` run proactively | All VP files + index in one burst |
| Canonical-surface rename residuals (retired identifier surviving in harness, story body, architecture prose) | Surfaced 6 separate times across consecutive passes, each in a file type NOT covered by the prior grep scope | full-scope grep eliminated the class entirely |

The 35-in-1 result is the clearest quantitative evidence: at the 1–2-per-pass discovery rate the loop was achieving before the manual sweep, that class alone would have consumed ~17–35 more LLM review pass-pairs before the corpus was clean — on anchors alone.

---

## What makes the loop non-terminating on these classes

The convergence gate definition (N consecutive clean passes over a stochastic full-corpus LLM pass; any material change resets) interacts badly with mechanically-enumerable classes:

1. Reviewer A finds instance K of a rename residual. The remediating burst fixes instance K but not sibling instance K+1 in a different file type (e.g., harness files were swept but architecture-rationale prose was not).
2. Reviewer B on the next pass finds instance K+1. Streak resets.
3. The fix for K+1 is applied but K+2 exists in yet another file type not covered.
4. Repeat.

This pattern was observed for the rename-residual class across 6 consecutive passes. Each individual fix was correct. The loop structure guaranteed recurrence because no single pass swept the entire corpus — each LLM reviewer has bounded context and tends to focus on the file type that showed the symptom.

---

## Minimum lint-coverage checklist

These checks are deterministic, require no LLM reasoning, and should all pass GREEN before the first adversarial pass of any phase:

1. **No TBD story anchors** — no active-owned contract may carry a TBD story-anchor field after story decomposition is complete.
2. **Canonical surface agreement (intra-document)** — every story's architecture-mapping table must agree with its own tasks/AC body on every enum variant set, struct field set, function signature, and method arity named in both sections.
3. **Canonical surface agreement (inter-document, bidirectional)** — for every type/function surface appearing in BOTH a story AC/task description AND an architecture document, the definition must agree bidirectionally. Both propagation directions must be checked.
4. **Index ↔ detail reconciliation** — every index file's count, version, and ownership cells must agree with the corresponding detail file's heading, frontmatter version, and ownership declaration.
5. **Cross-story prose-restatement agreement** — every "previous story intelligence" / "from story-N" prose block must be diffed against the canonical definitions in the module-decomposition and entity-definition documents. Any field/type/variant name present in the restatement but absent from the canonical definition (or vice versa) is a blocker.
6. **Dependency-table version/license agreement** — every library entry in any story's dependency table must match the canonical entry in the architecture's module-decomposition section.
7. **Canonical frontmatter conformance on ALL detail files** — run `validate-template-compliance` (the engine's own shipped skill) against every story, contract, and VP file before any pass is dispatched.
8. **Rename-propagation completeness (corpus-wide)** — after ANY canonical-surface rename or retirement, the residual grep must run across the ENTIRE corpus including architecture docs, rationale documents, assessment narratives, and VP prose blocks — not only harness/story/index files. This must execute as the closing step of any remediating burst before the pass can be declared clean.
9. **VP harness compilability** — for every VP proof-harness skeleton, mechanically check receiver form, arity, constructor existence, and enum-comparison form against the canonical signatures. A `cargo check`-style gate against a stub crate would catch this class mechanically.

---

## This unifies existing per-class requests

The following existing issues each address one defect class from the list above. This enhancement is the unifying proposal none of them states — a single mandatory pre-review layer that runs all checks at once, before any LLM pass is dispatched:

- #309 — TBD anchor / stale-reference class
- #299 — index ↔ detail reconciliation
- #287 — frontmatter conformance
- #216 — cross-document consistency
- #308 — per-finding CI job proliferation concern

---

## Response to the #308 tension

#308 raises a valid concern: if every finding that surfaces in a convergence pass spawns a new CI job, the CI job set becomes unbounded. This enhancement resolves that tension directly. The pre-review lint layer is:

- **Fixed and bounded** — the checklist above is finite and specified in advance, not grown by per-finding CI job creation.
- **Deterministic** — every check is a grep, schema validation, or structural comparison. No LLM reasoning. No false-negative variation across runs.
- **Runs once per cycle** — the lint gate runs once before the first LLM pass of each phase, not once per finding. It does not proliferate.

The per-finding CI job model (#308's concern) is orthogonal. The pre-review lint layer replaces the need for reactive per-class CI jobs by preventing the enumerable classes from ever reaching an LLM pass in the first place.

---

## Relationship to genuine semantic defects

This enhancement does NOT argue that the convergence loop is pure thrash. In the same pilot run, an early pass found two genuine build-blocking semantic defects (incompatible type-surface definitions across stories; a proof obligation with no valid owner) that survived ~9 prior structural passes. These require LLM-level cross-story semantic reasoning to find — no deterministic lint can catch a type-surface naming contradiction that spans story-ownership rules. The correct framing: a mandatory pre-review lint layer eliminates the cosmetic/mechanical thrash class entirely, freeing every LLM pass to focus on defects of exactly that semantic class.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.