BOHICA-LABS / BOHICA-LABS/vsdd-factory

bug(orchestrator): adversary L1 task prompt asks about details out of scope for opaque-string plumbing story — invites hallucinated findings

Open
#405 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

When orchestrator dispatches adversary L1 (impl-correctness) for a story whose scope is opaque-string plumbing (e.g., \"pass this address through without interpretation\"), a boilerplate L1 prompt that asks about \"hashing / derivation / endianness / normalization\" invites the adversary to invent findings about behavior the story explicitly disclaims.

## Concrete evidence (switchboard-blue Wave-6 Tranche B)

`S-BL.ROUTER-ADDR` scope: plumb a router address (opaque string) from config through PathTracker to Snapshot. No parsing, no hashing, no normalization — the story explicitly says \"opaque string.\"

Adversary L1 prompt across multiple passes asked about \"address parsing correctness / IPv6 handling / endianness of the derived key\" — none of which the story owns. This burned several passes on false-positive findings until Ruling-K clarified scope; and required test-writer to add comment obligations pinning the design (RULING-W6TB-K F-P4L2-03).

## Why it happens

Orchestrator's L1 dispatch prompt template is generic — it enumerates a broad L1 checklist. For \"transport-layer / plumbing\" stories the checklist is a superset of what applies. The adversary, having no way to know which checklist items are out-of-scope, dutifully investigates them and reports \"no evidence found for X\" as a HIGH finding.

## Proposed remediation

**Orchestrator dispatch:** derive an L1 sub-checklist from the story's declared scope tags. E.g., if `story.tags` includes `plumbing` or `opaque-transport`, drop \"parsing / normalization / derivation / endianness\" from the L1 checklist and add a leading clause: \"THIS STORY IS OPAQUE PLUMBING. Findings about the payload's internal structure are OUT OF SCOPE and must not be reported.\"

**Story template:** add a `scope_exclusions` frontmatter field listing behaviors the story explicitly does NOT own. Orchestrator injects these into every adversary dispatch.

**adversary policy:** any finding that names a behavior in `scope_exclusions` is auto-suppressed at ruling time.

## Cross-references

- #389 — preflight tuple accuracy (adjacent)
- #391 — mis-anchoring severity (this is the inverse: reviewer inventing an anchor)
- #278 — severity calibration drift (adjacent: this is calibration on scope, not severity)

## Severity

MEDIUM. Wastes passes and inflates finding counts; not a correctness defect but a discipline-erosion defect.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.