BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(product-owner+architect): holdout-scenario 'architect must evaluate' decision items don't route — surface as architect-decision tickets during Phase-1d

Open
#459 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

Holdout scenarios sometimes embed explicit architect-decision items in prose (e.g. \"the architect must evaluate whether X is sufficient or additional protection is required\"). These items are load-bearing — the answer determines whether the scenario passes or fails when evaluated in Phase-4. But the current Phase-1c/1d/2 workflow **does not extract these items and route them to the architect as tracked decisions**. They live silently in the scenario markdown until Phase-4 evaluation surfaces them as gate failures — 6 days and multiple phase transitions after they should have been resolved.

## Evidence — HS-002 Attack 3 on akey, 2026-07-03

Scenario `HS-002-llm-bypass-prevention.md`, produced by product-owner during Phase-1c (2026-06-27), Attack Vector 3 section:

> ## Test Execution — Attack Vector 3: Direct File Write (Threat Model Check)
>
> Attempt to write to the whitelist config file directly from a subprocess that has the same filesystem permissions as the akey process.
>
> Expected: This depends on ASM-001's enforcement mechanism. If OS file permissions are the only protection, this attack succeeds. **The architect must evaluate this attack vector and decide whether file permissions alone are sufficient or additional protection is required.**

The scenario notes below the acceptance criteria section escalate this:

> This holdout scenario is explicitly marked as HIGH impact in the risk register (R-001). It must be evaluated before Stage 1 ships. The architect's resolution of ASM-001 shapes which attack vectors are mitigated and which remain as accepted risks.

**What happened between 2026-06-27 (scenario written) and 2026-07-03 (Phase-4 evaluation):**
- Phase-1d adversarial spec convergence: 3+ clean passes on the specs, ASM-001 was not surfaced as unresolved (adversary reviews spec artifacts, not scenario prose containing decision items).
- Phase-2 story decomposition: BC-2.4.002 and BC-2.4.003 (the BCs HS-002 traces to) were decomposed into stories. Neither story pulled the ASM-001 decision forward as an architect input.
- Phase-3 TDD implementation (4 waves, ~15 stories, ~7 weeks of pipeline time): whitelist enforcement was implemented with load-time SHA-256 hash pinning of `.whitelist.hash` beside `whitelist.toml`, both files at `-rw-r--r--` in the same directory. Naive Attack-3 (write whitelist only) fires HashMismatch. Coordinated Attack-3 (write both files in lockstep) succeeds — daemon starts with wildcard auto-approve rule active.
- Phase-4 holdout evaluation (this session): scenario evaluator executed Attack-3 coordinated variant, ASM-001 was never resolved by the architect, DI-001 invariant is broken, HS-002 release-blocking failure condition is triggered.

Net: an architect-decision item that the scenario itself flagged as \"must be evaluated before Stage 1 ships\" was never actually scheduled for the architect to evaluate. It surfaced only at gate-check, 6 days late, forcing either a Wave-5 hardening PR or a documented threat-model amendment before Phase-4 can pass.

## Proposed shape

1. **Product-owner Phase-1c convention.** When writing a holdout scenario, any prose of the form \"the architect must evaluate…\", \"depends on the architect's resolution of…\", \"must be decided before…\" becomes a first-class decision item, extracted into a companion `decisions-required.md` in `.factory/holdout-scenarios/wave-scenarios/`.
2. **Phase-1d architect step.** During Phase-1d adversarial spec convergence, the orchestrator dispatches the architect against the `decisions-required.md` list before any adversary pass. Every listed decision returns either (a) a written resolution (ADR-style) that becomes a spec update, or (b) an accepted-risk entry with explicit rationale that becomes a documented amendment to the scenario.
3. **Adversary rubric addition.** The adversary is prompted to detect unresolved decision items in holdout scenarios during Phase-1d — a scenario that still contains \"the architect must evaluate…\" prose without a linked resolution is a spec-level finding.
4. **Phase-4 pre-check.** Before dispatching holdout-evaluator, the orchestrator scans wave-scenarios for unresolved decision items. Any hit blocks the Phase-4 dispatch until resolved — surfacing the gap at scenario-authoring time instead of at gate-check time.

## Adjacency to existing issues

Related but distinct from:
- **#435** (Fix Phase 2 spec propagation gap): #435 is about spec-side fixes not propagating to story body/AC prose. This issue is about decision items embedded in *scenario* prose not being extracted into the architect's queue in the first place. Different artifact class, different phase.
- **#422** (BC postconditions downstream of software boundary): related to holdout-scenario testability concerns, but #422 addresses ecosystem-observable postconditions in BCs. This issue addresses embedded architect-decision items in scenarios. Different symptoms.
- **#432** (no product-level defect register): tangentially related — if a defect register existed, unresolved architect-decisions could route through it. Different failure mode.

## Cost of the gap in this session

- 7 weeks of pipeline time between scenario authoring and defect surfacing.
- HS-002 Attack 3 is release-blocking per the scenario's own text. The gap forces either:
- A Wave-5 hardening PR to add real protection (HMAC of whitelist under a per-machine key, or append-only signed entries, etc.), pushing release, or
- An amended threat model that documents \"filesystem-write access = game over\" as accepted risk, requiring a scenario rewrite plus explicit human sign-off on the ASM-001 amendment.

Either way, the resolution happens at gate-check under pipeline pressure. The exact same decision made at Phase-1c-authoring time would have been made calmly with the whole design space open.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.