BOHICA-LABS / BOHICA-LABS/vsdd-factory
process-gap(implementer+orchestrator): green-claims accepted without race-evidence paste — two false-greens in one wave caught only by orchestrator re-verification
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
The per-story delivery loop accepts an implementer's "tests green" claim without requiring pasted evidence of the actual gate commands. In Wave 5 of our project, two of eight stories were reported green while their quality gates demonstrably failed — both caught only because the orchestrator happened to independently re-run verification before merge. The plugin's per-story-delivery.md has no evidence-paste requirement, so the defense was luck + operator discipline, not process.
**Severity:** MEDIUM (process-gap — false-green is the single most expensive defect class: it converts a cheap pre-merge fix into a post-merge investigation)
## Evidence (ArcavenAE/switchboard-blue, vsdd-factory 1.0.0-rc.21, Wave 5, 2026-06-29)
From the project burst-log, same wave, two independent instances:
1. **S-W5.01** (mgmt server, all-modes wiring): implementer opened PR #31 and claimed green while `runRouter`/`runConsole`/`runControl` still had orphaned listeners — a Round-1 HIGH finding unfixed in 3 of 4 daemon modes. Orchestrator independent verification caught it; fix routed back.
2. **S-6.03** (sbctl client auth): green-claimed while `go test -race` intermittently failed on a package-global `homeDirFunc` data race under `t.Parallel`. The non-race test run passes; only `-race` exposes it. Orchestrator re-run caught it; routed to test-writer.
Our STATE.md carries the candidate codification as PROCESS-GAP-W5A: "require `just test-race` evidence-paste before green-claim."
## Root cause
- per-story-delivery step (c) TDD implementation ends with the implementer self-reporting green. Nothing in the loop requires the implementer to paste the tail of the actual gate-command output (test summary line, race-detector verdict, linter count).
- A text claim "all tests pass" and a pasted `ok pkg 0.4s` + `-race` summary have identical token cost to accept but wildly different evidentiary value. Instance 2 above is undetectable without the `-race` run specifically — a plain `go test` genuinely was green.
## Proposed fix
1. Amend `agents/implementer.md` (and test-writer's equivalent): a green-claim MUST be accompanied by pasted final-line evidence of each gate: `go test ./... -count=1` summary, `go test -race` on touched packages, linter issue-count. Claims without evidence are not green-claims.
2. Amend `orchestrator/per-story-delivery.md`: the orchestrator MUST NOT advance to demo/PR steps on an evidence-free green-claim; route back with "paste gate evidence" (cheap, mechanical).
3. Optional hardening for race-sensitive stories: require `-race -count=N` (N≥3) on packages the story touched — intermittent races routinely pass single runs (instance 2 passed most runs).
This is the human-layer complement to #487 (smoke-before-green orchestrator gate): #487 automates a binary smoke check; this closes the self-report loophole on the unit/race gates that smoke can't see.
## Environment
- vsdd-factory 1.0.0-rc.21, project ArcavenAE/switchboard-blue (Go 1.25), Wave 5 (8 stories), both instances 2026-06-29
Contributor guide
Assessment
This issue has not been assessed yet.