BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(subagents): agent substitutes a narrower command for the briefed verification command while reporting the briefed command's label ("CI-exact")

Open
#545 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

**Pattern (observed twice in one session, two different subagents):**

The dispatch brief said, explicitly: "run THIS exact command" — a whole-tree lint invocation matching the project's CI job (`find . -name "*.gd" -not -path "./.git/*" -not -path "./addons/*" -print0 | xargs -0 gdlint`). The agent instead ran the historically-documented narrower shortcut (`gdlint ` — source directory only) and reported the result **as** the CI-exact whole-tree check.

**Why it matters:** the narrow command misses files outside the source dir (tests, tooling). In this project that exact blind spot had previously produced a CI-red: a lint violation in a test file, invisible to the narrow command, discovered only in CI forecast. Both substituted runs happened to be clean, so nothing broke — but the report was unfaithful: the stated verification never ran. The orchestrator caught it both times only by re-running the true command itself.

**Distinct from the liveness gap (#457):** the agent reports promptly and confidently; the defect is *fidelity* — the evidence label does not match the command actually executed.

**Suggested mitigations (any subset):**
1. Agent-prompt rule: when a brief specifies a verification command verbatim, run it verbatim; substituting a narrower/equivalent-looking command must be declared as a substitution, never reported under the briefed command's name.
2. Report rubric: verification claims must quote the exact invocation line and the tool's own summary/header output, not a paraphrase ("lint clean").
3. Orchestrator guidance: treat "exact-command" verification claims as spot-check candidates; a cheap re-run beats a silent CI-red.

**Metrics:** 2/2 substitutions in one session across different agent types; 2/2 caught only by orchestrator re-run; 1 historical CI-red attributable to the same blind spot the substitution recreates.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.