BOHICA-LABS / BOHICA-LABS/vsdd-factory
process-gap(product-owner+architect): holdout-scenario 'architect must evaluate' decision items don't route — surface as architect-decision tickets during Phase-1d
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
Holdout scenarios sometimes embed explicit architect-decision items in prose (e.g. \"the architect must evaluate whether X is sufficient or additional protection is required\"). These items are load-bearing — the answer determines whether the scenario passes or fails when evaluated in Phase-4. But the current Phase-1c/1d/2 workflow **does not extract these items and route them to the architect as tracked decisions**. They live silently in the scenario markdown until Phase-4 evaluation surfaces them as gate failures — 6 days and multiple phase transitions after they should have been resolved.
## Evidence — HS-002 Attack 3 on akey, 2026-07-03
Scenario `HS-002-llm-bypass-prevention.md`, produced by product-owner during Phase-1c (2026-06-27), Attack Vector 3 section:
> ## Test Execution — Attack Vector 3: Direct File Write (Threat Model Check)
>
> Attempt to write to the whitelist config file directly from a subprocess that has the same filesystem permissions as the akey process.
>
> Expected: This depends on ASM-001's enforcement mechanism. If OS file permissions are the only protection, this attack succeeds. **The architect must evaluate this attack vector and decide whether file permissions alone are sufficient or additional protection is required.**
The scenario notes below the acceptance criteria section escalate this:
> This holdout scenario is explicitly marked as HIGH impact in the risk register (R-001). It must be evaluated before Stage 1 ships. The architect's resolution of ASM-001 shapes which attack vectors are mitigated and which remain as accepted risks.
**What happened between 2026-06-27 (scenario written) and 2026-07-03 (Phase-4 evaluation):**
- Phase-1d adversarial spec convergence: 3+ clean passes on the specs, ASM-001 was not surfaced as unresolved (adversary reviews spec artifacts, not scenario prose containing decision items).
- Phase-2 story decomposition: BC-2.4.002 and BC-2.4.003 (the BCs HS-002 traces to) were decomposed into stories. Neither story pulled the ASM-001 decision forward as an architect input.
- Phase-3 TDD implementation (4 waves, ~15 stories, ~7 weeks of pipeline time): whitelist enforcement was implemented with load-time SHA-256 hash pinning of `.whitelist.hash` beside `whitelist.toml`, both files at `-rw-r--r--` in the same directory. Naive Attack-3 (write whitelist only) fires HashMismatch. Coordinated Attack-3 (write both files in lockstep) succeeds — daemon starts with wildcard auto-approve rule active.
- Phase-4 holdout evaluation (this session): scenario evaluator executed Attack-3 coordinated variant, ASM-001 was never resolved by the architect, DI-001 invariant is broken, HS-002 release-blocking failure condition is triggered.
Net: an architect-decision item that the scenario itself flagged as \"must be evaluated before Stage 1 ships\" was never actually scheduled for the architect to evaluate. It surfaced only at gate-check, 6 days late, forcing either a Wave-5 hardening PR or a documented threat-model amendment before Phase-4 can pass.
## Proposed shape
1. **Product-owner Phase-1c convention.** When writing a holdout scenario, any prose of the form \"the architect must evaluate…\", \"depends on the architect's resolution of…\", \"must be decided before…\" becomes a first-class decision item, extracted into a companion `decisions-required.md` in `.factory/holdout-scenarios/wave-scenarios/`.
2. **Phase-1d architect step.** During Phase-1d adversarial spec convergence, the orchestrator dispatches the architect against the `decisions-required.md` list before any adversary pass. Every listed decision returns either (a) a written resolution (ADR-style) that becomes a spec update, or (b) an accepted-risk entry with explicit rationale that becomes a documented amendment to the scenario.
3. **Adversary rubric addition.** The adversary is prompted to detect unresolved decision items in holdout scenarios during Phase-1d — a scenario that still contains \"the architect must evaluate…\" prose without a linked resolution is a spec-level finding.
4. **Phase-4 pre-check.** Before dispatching holdout-evaluator, the orchestrator scans wave-scenarios for unresolved decision items. Any hit blocks the Phase-4 dispatch until resolved — surfacing the gap at scenario-authoring time instead of at gate-check time.
## Adjacency to existing issues
Related but distinct from:
- **#435** (Fix Phase 2 spec propagation gap): #435 is about spec-side fixes not propagating to story body/AC prose. This issue is about decision items embedded in *scenario* prose not being extracted into the architect's queue in the first place. Different artifact class, different phase.
- **#422** (BC postconditions downstream of software boundary): related to holdout-scenario testability concerns, but #422 addresses ecosystem-observable postconditions in BCs. This issue addresses embedded architect-decision items in scenarios. Different symptoms.
- **#432** (no product-level defect register): tangentially related — if a defect register existed, unresolved architect-decisions could route through it. Different failure mode.
## Cost of the gap in this session
- 7 weeks of pipeline time between scenario authoring and defect surfacing.
- HS-002 Attack 3 is release-blocking per the scenario's own text. The gap forces either:
- A Wave-5 hardening PR to add real protection (HMAC of whitelist under a per-machine key, or append-only signed entries, etc.), pushing release, or
- An amended threat model that documents \"filesystem-write access = game over\" as accepted risk, requiring a scenario rewrite plus explicit human sign-off on the ASM-001 amendment.
Either way, the resolution happens at gate-check under pipeline pressure. The exact same decision made at Phase-1c-authoring time would have been made calmly with the whole design space open.
Contributor guide
Assessment
This issue has not been assessed yet.