BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(adversary): probe variance, not pass count, drives convergence quality — hostile-input structure axis and path-component enumeration missing from methodology

Open
#462 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

Per-story adversarial convergence quality is dominated by **probe variance across passes**, not pass count. In a 15-pass fresh-context convergence loop on one story, crash-class defects surfaced at passes 9 and 11 — *after* earlier passes (including a CLEAN one) had walked the same code. The late catches came exclusively from passes whose dispatch prompts steered toward input surfaces no earlier pass had tried. Two generalizable methodology gaps fell out, plus one termination-heuristic caveat.

## Evidence (single story, 15 fresh-context passes, private pilot; identifiers scrubbed)

Verdict sequence: NC,NC,NC,NC,**C**,NC,C-minor,NC,**NC(CRITICAL)**,C-minor,**NC(2×CRITICAL)**,NC,C,C,C

- Pass 9 CRITICAL: a non-Dictionary element inside a JSON `steps` array crashed an unchecked cast — the harness hung with no exit code. Eight prior passes had probed "hostile plan values" but only ever varied *values* (typo'd action names, bad top-level shape) — never **element shape inside arrays**.
- Pass 11 2×CRITICAL: non-String `story_id` crashed a typed assignment (process hang); non-String `label` / non-Dict `mouse_click` corrupted the output manifest with empty `{}` step objects. Ten prior passes had probed **path traversal** — but only on the *leaf* path component (screenshot label), never enumerating **every user-controlled path component** (`story_id` reached `path_join` unchecked).

## Gap 1 — hostile-input probes must vary STRUCTURE, not just values

"Hostile plan values" repeated across passes converges on a value-fuzzing rut. The high-yield axis was type/shape: wrong-typed fields, wrong-shaped container elements, valid-JSON-wrong-topology. Suggested adversary-template axis for any JSON/config-consuming surface: per field and per container, probe {wrong type, wrong element shape, empty, missing, duplicate-key}.

## Gap 2 — path-traversal probes must enumerate every user-controlled path component

The reviewer that found the `story_id` escape said it directly: earlier remediation "closed on label safety without extending to story_id." Any finding of the form "component X is now sanitized" should trigger an enumeration probe: list every other component that reaches path construction and verify each.

## Termination caveat (relates to convergence-termination heuristics, e.g. #344)

A cosmetic-decay signal ("recent findings are all doc-polish") is only trustworthy if probe angles are still varying. In this loop, the finding stream looked cosmetic at passes 5–8; passes 9 and 11 then produced three CRITICALs — because their prompts forced novel angles. A terminal-round judgment should require BOTH finding-decay AND evidence that recent passes attacked materially different surfaces (the orchestrator can carry a probes-tried list in dispatch prompts and require each pass to favor untried angles — that is what produced the late catches here).

## What worked (adoptable pattern)

The orchestrator's dispatch prompt for each pass carried: (a) a cumulative "prior passes covered: ..." list, and (b) an explicit "favor NOVEL high-yield angles; still-untried ideas: ..." menu. Passes given this steering found the defects the un-steered passes missed. Cost: a few prompt lines per pass.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.