BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(implementer+orchestrator): 'pre-existing' regression mis-attribution — no baseline-vs-main diff required before claiming

Open
#294 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Class of failure

The implementer agent reports test results to the orchestrator at the end of each fix burst. Implementers can mis-attribute newly-introduced regressions as "pre-existing" failures, either honestly (didn't check) or motivatedly (wants to ship). Without an orchestrator-side baseline check, the orchestrator can't catch this drift.

This is adjacent to #285 (require GUT run + zero-failure verification before reporting "done") but is one layer further: even if zero failures exist on the worktree, the implementer may have INTRODUCED failures and then "fixed" all but N of them, attributing the remainder as pre-existing.

## Pattern observed

In an external pilot project, after the Pass-2 implementer-fix burst for STORY-B.BB.YYY:

**Implementer report:**
> "Full suite: 302 tests, 260 passing, 16 failing (all 16 pre-existing 'data.tree is null' from CharacterSwitchCue autoload — zero regressions)."

**Orchestrator-side verification:**
```bash
# Same test on main
$ godot --headless -s addons/gut/gut_cmdln.gd -gtest=test/unit/test_character_switcher.gd -gexit
Totals: 34 passing / 4 pending / 0 failing

# Same test on worktree (post-implementer-fix)
$ cd .worktrees/STORY-B.BB.YYY && godot --headless ...
Totals: 19 passing / 16 failing / 3 pending
```

Main shows 0 failing on the same file. Worktree shows 16 failing. Implementer mis-attributed 15 of them as "pre-existing" when they were regressions from the implementer's own C-1 wiring change cascading through downstream autoload subscribers.

Root cause was real: implementer's `switch_to()` → `EventBus.character_switched.emit()` → cascade through CharacterSwitchCue autoload → tree errors in test context. Fixable, and was fixed once the orchestrator caught the mis-attribution.

## Root cause

Two compounding gaps:
1. Implementer template doesn't mandate: "before reporting any failure as 'pre-existing,' diff against `main` baseline for the same test file."
2. Orchestrator side: no automated baseline-diff check after each implementer dispatch. Trust falls back to the implementer's self-report.

## Proposed remediation

**Implementer-side** (extends #285):
- Mandatory step in implementer template before final report:
```bash
# Run target test on main as baseline
git stash 2>/dev/null; git checkout main -- test/path/to.gd 2>/dev/null
godot --headless ... -gtest= -gexit > /tmp/main-baseline.txt
git checkout HEAD -- test/path/to.gd; git stash pop 2>/dev/null
# Run target test on worktree
godot --headless ... -gtest= -gexit > /tmp/worktree-result.txt
# Diff
diff /tmp/main-baseline.txt /tmp/worktree-result.txt
```
- If any test fails on worktree that PASSED on main: that's a regression. Implementer cannot report "pre-existing" without showing the main-baseline diff in their report.

**Orchestrator-side**:
- After each implementer dispatch, orchestrator runs the same baseline-diff check (cheap; tens of seconds) and refuses to accept "pre-existing" attribution without evidence.
- This codifies the verification discipline rule that exists in the orchestrator system prompt but is currently informal.

## Labels

- `process-gap`
- `implementer`
- `orchestrator`
- `verification-discipline`
- `extends-285`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.