BOHICA-LABS / BOHICA-LABS/vsdd-factory
enhancement(implementer): require GUT run + zero-failure verification before reporting "done"
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
The `implementer` agent reports success based on its own reads of test-tool output, but does not consistently run a final verification suite and parse failures before declaring "done". Multiple times in long-running deliveries, implementers have reported "all tests pass" while a final independent verification revealed failures that the implementer either didn't run, ran and misread, or ran and chose not to report.
## Observed pattern
I delegate a fix burst to an implementer with explicit "all 220+ existing passing tests must still pass, run GUT, confirm zero failing" constraints. Implementer report comes back: "Status: DONE. Final GUT totals: 250/225/0/25." Independent verification (running the same GUT command myself, parsing the log) finds either:
- A real failing test the implementer didn't see
- A misread of GUT marker characters (`*` interpreted as "failed" or vice versa)
- A test the implementer thought was acceptable but actually broke a regression contract
In one concrete case across stories I've delivered, this happened despite explicit "STOP and report if any test fails" instructions. The implementer's confidence was high; the verification was missing.
## Why this matters
- Failure to verify means failures escape to the orchestrator who relies on the report.
- Orchestrators that re-verify catch ~30-50% of these (rough estimate from my work).
- Implementers that re-verify their own work catch them at the source.
- The cost asymmetry is large: an implementer self-verifying is 1 GUT run; an orchestrator catching it is 1 GUT run + dispatch + re-fix + re-verify cycle.
## Suggested mitigation
Update the `implementer` agent prompt to include an **explicit final-verification protocol**:
1. After the last commit, run the project's test suite headlessly with explicit flags
2. Parse the output for the totals block (`Tests N / Passing P / Failing F / Pending S`)
3. If `Failing > 0`: STOP and report which tests failed, do NOT report "done"
4. Otherwise: include the verified totals block verbatim in the exit report
This converts the implementer's claim "tests pass" from inference to evidence.
## Related
- #267 (state-manager source-of-truth validation) — sibling pattern: agent reports without ground-truth check
- #283 (convergence-loop hardening) — orchestrator-side perspective on the same failure mode (orchestrator must re-verify because implementer didn't)
Contributor guide
Assessment
This issue has not been assessed yet.