BOHICA-LABS / BOHICA-LABS/vsdd-factory
bug(phase-4 gate): hard-coded "GPT-5.4, not Claude" criterion with fail_action: block is unsatisfiable in the plugin substrate
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Symptom
The Phase-4 holdout gate (`workflows/phases/phase-4-holdout-evaluation.lobster:28-38`, duplicated in `workflows/greenfield.lobster:~1039`) hard-codes:
```yaml
- name: holdout-gate
type: gate
gate:
criteria:
- "Holdout evaluator used different model family (GPT-5.4, not Claude)"
...
fail_action: block
```
The `holdout-evaluator` is a Claude Code subagent. In the plugin substrate there is no control surface to dispatch it on a non-Anthropic model (#417 documents this as platform-inherited: model selection is host-owned). So **the first gate criterion is unsatisfiable as literally written, and `fail_action: block` means a strictly-read Phase 4 can never pass.**
In practice orchestrators satisfy the *intent* via intra-Claude diversity (e.g. opus evaluator vs sonnet builders, per BC-5.39.001 practice) and either gloss the criterion or mark it waived — i.e., the gate is passed by reinterpretation, not by satisfaction. That's the same self-ratification pressure #260 documents, but here the workflow text itself forces it.
Same hard-coded assumption appears in preflights (`greenfield.lobster:579/595`, `brownfield.lobster:37/50`, `planning.lobster:38` — "All 3 model families reachable (Claude, GPT-5.4, Gemini) REQUIRED") and Phase-5 prose. Those preflights are equally unsatisfiable on a stock Claude Code install with no LiteLLM router.
## Why this isn't just #410/#417
#410/#417 track the platform-envelope decision. This issue is narrower and code-fixable **today**: regardless of the replatform/amendment decision, the shipped workflow files should not contain a blocking gate criterion that no supported install can satisfy. The criterion should express the *invariant* (evaluator model ≠ any builder/implementer model for this wave, verified via dispatch telemetry per #468) with the cross-vendor ideal as a conditional upgrade when a router is configured.
## Suggested fix
1. Reword the gate criterion to: "Evaluator ran on a model distinct from all Phase-3 implementer/test-writer models for the wave (verify via dispatch record, not agent self-report — see #468). Cross-vendor family (e.g. GPT/Gemini) REQUIRED only when a multi-vendor router is configured; otherwise intra-vendor tier diversity + information-asymmetry wall is the documented fallback."
2. Same treatment for the LLM-preflight criteria in greenfield/brownfield/planning lobsters (`REQUIRED` → conditional on router config).
3. Record the actual evaluator model id in the evaluation report frontmatter so the gate is auditable.
## Environment / evidence
vsdd-factory 1.0.0-rc.21, greenfield on ArcavenAE/akey, Phase-4 dispatch 2026-07-04. No LiteLLM/multi-vendor router configured; holdout-evaluator can only be a Claude model.
## Related
- #410, #417 (platform-inherited model routing family)
- #468 (dispatched-model fidelity / self-report unreliability — the verification mechanism this gate needs)
- #260 (gate self-ratification pressure)
Suggested labels: `bug`, `workflows`, `phase-4`, `gates`
Contributor guide
Assessment
This issue has not been assessed yet.