BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(orchestrator+adversary): remediation diffs get no deterministic targeted re-review — regressions caught only probabilistically by the next full pass

Open
#322 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Class of failure

The convergence loop's only check on a **remediation diff** is the *next* full fresh-context adversary/consistency-validator pass. That pass is a whole-corpus review, not a targeted re-review of the lines just changed. So a regression introduced *by the remediation itself* is caught only **probabilistically** — it depends on whether the next stochastic full pass happens to traverse those exact surfaces before it emits a verdict (and, per #306, before it compacts).

This is the structural reason a convergence streak can ship **two consecutive partial-fix passes**: pass N remediates defect X but introduces stale-binding Y; pass N+1 (fresh context, full corpus) happens to find Y and resets the streak; pass N+1's remediation of Y introduces Z; and so on. Each individual pass is "clean by its own lights"; the loop converges slower than the pass count suggests, and the gate's reliability is a function of review luck, not a deterministic property.

Relationship to existing issues (this is the gap *between* them, not a dup):
- #299 proposes a guard for one specific defect class (duplicated-value set-equality). Necessary but per-class.
- #277 / #310 cover derived-count drift; #309 covers lint-class gates invisible to ground truth. All are *specific* surfaces.
- #283 Gap 2 surfaces convergence *trajectory observability* (continue/stop intuition) — measurement, not a gate.
- #222 documents the *rationale* for "3 clean passes"; this issue is about what "clean" is actually testing.

None of them make the **remediation diff itself** a first-class review object.

## Suggested direction

Add a deterministic, cheap **post-remediation self-check step** that runs *before* the next full adversarial pass, scoped to what the remediation touched:

1. **Diff-scoped binding sweep.** For every identifier re-anchored in the remediation (`X: A → B`), grep the whole corpus for residual `X → A` and fail if any survive (excluding immutable changelog prose). Generalizes #299's guard to "whatever this diff just changed."
2. **Diff-scoped derived-field recompute check.** If the diff added/removed primary content (BC, VP, story), assert the relevant counts/totals were recomputed (the #277/#310 surfaces) — but triggered *by the diff*, not by a separate validator pass.
3. **Surface the result as gate input.** The orchestrator attaches this self-check's output to the next adversary prompt as ground truth (same mechanism #298/#309 propose for the test/lint tally).

The point is not a new heavyweight reviewer — it's making the remediation step *deterministically self-verifying for the class of regressions remediations actually introduce*, so the next full pass is confirming convergence rather than being the first and only thing that might catch a self-inflicted regression.

## Environment

- Plugin: vsdd-factory `1.0.0-rc.21`
- Observed: orchestrator-driven Phase-1 spec convergence, fresh-context `adversary` + `consistency-validator` pair per pass, "3 consecutive clean passes" gate. Two consecutive passes each shipped (and the following pass caught) a partial-fix regression in spec cross-references.

(No private project content included, per #261.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.