BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(adversary+orchestrator): read-only adversary cannot execute the test suite, so per-story convergence trusts an unverifiable test tally

Open
#298 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Class of failure

The `adversary` agent is intentionally read-only (Read/Grep/Glob; no Bash/exec) for perspective independence. A side effect: during per-story adversarial convergence, the adversary **cannot run the test suite**. When it needs to confirm a claim like "394 tests / 366 passing / 0 failing," it can only (a) take the orchestrator's stated tally on faith, or (b) infer indirectly from static analysis (counting test scripts on disk, checking referenced symbols exist).

Neither is a true verification. A genuine green/red state — including the failure mode where a parse error silently drops scripts so "0 failing" is misleading — is invisible to the adversary.

## Pattern observed (external pilot, ftc-blue)

Across 10 fresh-context adversarial passes on one story, every pass had to caveat its verdict with language like: "I am read-only (no Bash) so I could not execute GUT; the '0 failing' claim is consistent with the 14 scripts present but not independently confirmed." The only thing that made convergence trustworthy was the **orchestrator independently running GUT after every pass** and at the convergence boundary — an out-of-band mitigation that isn't mandated anywhere.

## Why it matters

- A core convergence gate (the test tally) is asserted by the party being reviewed (orchestrator/implementer) and cannot be checked by the reviewer (adversary).
- If the orchestrator skips its own independent run, a convergence "clean pass" can rest on an unverified or stale test count.
- This compounds with the script-count-drop masking failure (see #294 and its addendum): the adversary is exactly the actor you'd want to catch a dropped-script regression, and it structurally can't.

## Proposed remediation (pick one)

1. **Orchestrator-attaches-evidence (preferred, no tooling change):** formalize in the per-story-delivery / orchestrator template that the orchestrator MUST run the test suite and attach the raw totals block (and script count) to each adversarial-pass dispatch prompt, so the adversary verifies against attached evidence rather than a bare claim.
2. **Verified-results artifact:** have a trusted step write a signed/timestamped test-results artifact (totals + script list) that the read-only adversary reads and checks for internal consistency (script count == expected, 0 failing, no "dropped script" markers).
3. Keep the adversary read-only (do NOT give it exec) — the fix is about supplying verifiable evidence, not loosening isolation.

## Severity
MEDIUM — convergence soundness; mitigated today only by orchestrator diligence.

## Cross-ref
Locally tracked as PG-3 in the pilot's cycle lessons. Related: #294, #285.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.