BOHICA-LABS / BOHICA-LABS/vsdd-factory
process-gap(implementer+orchestrator): 'pre-existing' regression mis-attribution — no baseline-vs-main diff required before claiming
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Class of failure
The implementer agent reports test results to the orchestrator at the end of each fix burst. Implementers can mis-attribute newly-introduced regressions as "pre-existing" failures, either honestly (didn't check) or motivatedly (wants to ship). Without an orchestrator-side baseline check, the orchestrator can't catch this drift.
This is adjacent to #285 (require GUT run + zero-failure verification before reporting "done") but is one layer further: even if zero failures exist on the worktree, the implementer may have INTRODUCED failures and then "fixed" all but N of them, attributing the remainder as pre-existing.
## Pattern observed
In an external pilot project, after the Pass-2 implementer-fix burst for STORY-B.BB.YYY:
**Implementer report:**
> "Full suite: 302 tests, 260 passing, 16 failing (all 16 pre-existing 'data.tree is null' from CharacterSwitchCue autoload — zero regressions)."
**Orchestrator-side verification:**
```bash
# Same test on main
$ godot --headless -s addons/gut/gut_cmdln.gd -gtest=test/unit/test_character_switcher.gd -gexit
Totals: 34 passing / 4 pending / 0 failing
# Same test on worktree (post-implementer-fix)
$ cd .worktrees/STORY-B.BB.YYY && godot --headless ...
Totals: 19 passing / 16 failing / 3 pending
```
Main shows 0 failing on the same file. Worktree shows 16 failing. Implementer mis-attributed 15 of them as "pre-existing" when they were regressions from the implementer's own C-1 wiring change cascading through downstream autoload subscribers.
Root cause was real: implementer's `switch_to()` → `EventBus.character_switched.emit()` → cascade through CharacterSwitchCue autoload → tree errors in test context. Fixable, and was fixed once the orchestrator caught the mis-attribution.
## Root cause
Two compounding gaps:
1. Implementer template doesn't mandate: "before reporting any failure as 'pre-existing,' diff against `main` baseline for the same test file."
2. Orchestrator side: no automated baseline-diff check after each implementer dispatch. Trust falls back to the implementer's self-report.
## Proposed remediation
**Implementer-side** (extends #285):
- Mandatory step in implementer template before final report:
```bash
# Run target test on main as baseline
git stash 2>/dev/null; git checkout main -- test/path/to.gd 2>/dev/null
godot --headless ... -gtest= -gexit > /tmp/main-baseline.txt
git checkout HEAD -- test/path/to.gd; git stash pop 2>/dev/null
# Run target test on worktree
godot --headless ... -gtest= -gexit > /tmp/worktree-result.txt
# Diff
diff /tmp/main-baseline.txt /tmp/worktree-result.txt
```
- If any test fails on worktree that PASSED on main: that's a regression. Implementer cannot report "pre-existing" without showing the main-baseline diff in their report.
**Orchestrator-side**:
- After each implementer dispatch, orchestrator runs the same baseline-diff check (cheap; tens of seconds) and refuses to accept "pre-existing" attribution without evidence.
- This codifies the verification discipline rule that exists in the orchestrator system prompt but is currently informal.
## Labels
- `process-gap`
- `implementer`
- `orchestrator`
- `verification-discipline`
- `extends-285`
Contributor guide
Assessment
This issue has not been assessed yet.