BOHICA-LABS / BOHICA-LABS/vsdd-factory

adversary: corroboration-inversion — fresh-context findings quote evidence that contradicts actual file contents

Open
#425 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

Fresh-context adversary sub-agent can produce high-severity findings whose cited evidence is INVERTED from the actual source. This is not phantom-symbol drift (#399) — this is phantom-evidence drift: the symbol exists at the cited file:line, but the adversary's quoted content contradicts what is on disk. The adversary asserts "line X does A" when line X verifiably does the opposite of A.

## Observed pattern (abstract)

A well-formed story with a recently-ratified BC (audit-ordering / CAS-first / fail-closed contract) was implemented correctly and had all integration tests passing green. On fresh-context Pass-1:

- Adversary returned 15 findings (4 CRIT / 5 HIGH / 5 MED / 3 LOW).
- On triage against actual HEAD source: **7 of 15 findings (~47%) were inverted-evidence hallucinations.**
- Specifically:
- Finding cites 'line N calls method A' → line N calls sibling method B (with different semantics that exactly satisfy the invariant the finding claims is violated).
- Finding claims 'doc-comment declares ordering X' → doc-comment declares ordering Y verbatim.
- Finding declares 'test WILL FAIL' → target test passes green in immediate re-run.
- Remaining 8 findings included 2 legitimate doc-hygiene issues, 1 legitimate scope question needing product-owner ruling, and 5 minor observability improvements.

## Distinct from related issues

- **#399 (phantom-API drift):** spec cites a symbol that doesn't exist → adversary passes CLEAN. This variant: symbol exists and code is correct → adversary invents contradictory evidence.
- **#405 (adversary asked about details out of scope):** adversary invited to hallucinate by prompt scope. This variant: prompt scope is correct; adversary hallucinates against the correct target.
- **#18-equivalent (adversary stalls on large-context corroboration):** adversary fails to corroborate. This variant: adversary produces confident anti-corroboration.

## Suggested remediation

Two layers, cheap:

### 1. Structured-finding schema

Require adversary to emit each finding in structured shape:
\`\`\`json
{
\"severity\": \"CRIT\" | \"HIGH\" | \"MED\" | \"LOW\",
\"file\": \"crates/foo/src/bar.rs\",
\"line\": 293,
\"evidence_quote_verbatim\": \"self.queue.remove(request.id).await;\",
\"invariant_violated\": \"BC-N.M.P POST-Q ...\",
\"suggested_fix\": \"...\"
}
\`\`\`

The \`evidence_quote_verbatim\` must be a byte-exact substring of the file (no paraphrase). System prompt tells the adversary this is machine-verified.

### 2. Post-adversary evidence-verification pass

Before dispatching remediation, orchestrator (or lightweight validator sub-agent) executes for every finding with severity ≥ HIGH:

\`\`\`
Read(file, line-2..line+5)
assert evidence_quote_verbatim in window
\`\`\`

Findings failing verification are tagged HALLUCINATED and NOT dispatched to implementer. They're surfaced to the human orchestrator for manual triage.

## Impact / downstream cost

- Convergence counter poisoning: false CRIT/HIGH findings reset 3-CLEAN to 0/3, burning subsequent adversary passes on findings that can't be remediated (because the code already does the right thing).
- Wasted implementer bursts: if orchestrator dispatches remediation without triage, implementer either reports 'no change needed' (best case) or, worse, is nudged into REMOVING correct code because the finding's rationale sounds plausible.
- Manual grep-triage: orchestrator has to Read every cited file:line by hand to distinguish real findings from hallucinated ones. At project cadence (~1-2 stories/day, 3+ fresh passes per story) this is on the order of 30-90 minutes per hallucination event. Multiplied across a dozen-story wave, aggregate cost is high.

## Why fresh-context makes this WORSE, not better

Fresh-context is supposed to reduce hallucination by starving the model of contaminating priors. But when the target artifact is complex (multi-hundred-line src file with intricate ordering invariants), the fresh model has to construct its mental model from a cold read. Under cognitive load it can:

- Read line X, form initial hypothesis about ordering.
- Continue reading and update the model.
- When emitting the finding, retrieve the initial hypothesis rather than the updated one → cite line X's actual content as evidence for the OPPOSITE claim.

The structured-schema + post-verification approach above closes this failure mode without requiring model-level fixes.

## Willing to share

Anonymized excerpts of the 7 hallucinated findings + corresponding actual source line contents available on request — useful as calibration data for prompt engineering or eval harness.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.