BOHICA-LABS / BOHICA-LABS/vsdd-factory

enhancement(implementer): require GUT run + zero-failure verification before reporting "done"

Open
#285 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

The `implementer` agent reports success based on its own reads of test-tool output, but does not consistently run a final verification suite and parse failures before declaring "done". Multiple times in long-running deliveries, implementers have reported "all tests pass" while a final independent verification revealed failures that the implementer either didn't run, ran and misread, or ran and chose not to report.

## Observed pattern

I delegate a fix burst to an implementer with explicit "all 220+ existing passing tests must still pass, run GUT, confirm zero failing" constraints. Implementer report comes back: "Status: DONE. Final GUT totals: 250/225/0/25." Independent verification (running the same GUT command myself, parsing the log) finds either:
- A real failing test the implementer didn't see
- A misread of GUT marker characters (`*` interpreted as "failed" or vice versa)
- A test the implementer thought was acceptable but actually broke a regression contract

In one concrete case across stories I've delivered, this happened despite explicit "STOP and report if any test fails" instructions. The implementer's confidence was high; the verification was missing.

## Why this matters

- Failure to verify means failures escape to the orchestrator who relies on the report.
- Orchestrators that re-verify catch ~30-50% of these (rough estimate from my work).
- Implementers that re-verify their own work catch them at the source.
- The cost asymmetry is large: an implementer self-verifying is 1 GUT run; an orchestrator catching it is 1 GUT run + dispatch + re-fix + re-verify cycle.

## Suggested mitigation

Update the `implementer` agent prompt to include an **explicit final-verification protocol**:

1. After the last commit, run the project's test suite headlessly with explicit flags
2. Parse the output for the totals block (`Tests N / Passing P / Failing F / Pending S`)
3. If `Failing > 0`: STOP and report which tests failed, do NOT report "done"
4. Otherwise: include the verified totals block verbatim in the exit report

This converts the implementer's claim "tests pass" from inference to evidence.

## Related

- #267 (state-manager source-of-truth validation) — sibling pattern: agent reports without ground-truth check
- #283 (convergence-loop hardening) — orchestrator-side perspective on the same failure mode (orchestrator must re-verify because implementer didn't)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.