BOHICA-LABS / BOHICA-LABS/vsdd-factory

bug(phase-4 gate): hard-coded "GPT-5.4, not Claude" criterion with fail_action: block is unsatisfiable in the plugin substrate

Open
#474 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Symptom

The Phase-4 holdout gate (`workflows/phases/phase-4-holdout-evaluation.lobster:28-38`, duplicated in `workflows/greenfield.lobster:~1039`) hard-codes:

```yaml
- name: holdout-gate
type: gate
gate:
criteria:
- "Holdout evaluator used different model family (GPT-5.4, not Claude)"
...
fail_action: block
```

The `holdout-evaluator` is a Claude Code subagent. In the plugin substrate there is no control surface to dispatch it on a non-Anthropic model (#417 documents this as platform-inherited: model selection is host-owned). So **the first gate criterion is unsatisfiable as literally written, and `fail_action: block` means a strictly-read Phase 4 can never pass.**

In practice orchestrators satisfy the *intent* via intra-Claude diversity (e.g. opus evaluator vs sonnet builders, per BC-5.39.001 practice) and either gloss the criterion or mark it waived — i.e., the gate is passed by reinterpretation, not by satisfaction. That's the same self-ratification pressure #260 documents, but here the workflow text itself forces it.

Same hard-coded assumption appears in preflights (`greenfield.lobster:579/595`, `brownfield.lobster:37/50`, `planning.lobster:38` — "All 3 model families reachable (Claude, GPT-5.4, Gemini) REQUIRED") and Phase-5 prose. Those preflights are equally unsatisfiable on a stock Claude Code install with no LiteLLM router.

## Why this isn't just #410/#417

#410/#417 track the platform-envelope decision. This issue is narrower and code-fixable **today**: regardless of the replatform/amendment decision, the shipped workflow files should not contain a blocking gate criterion that no supported install can satisfy. The criterion should express the *invariant* (evaluator model ≠ any builder/implementer model for this wave, verified via dispatch telemetry per #468) with the cross-vendor ideal as a conditional upgrade when a router is configured.

## Suggested fix

1. Reword the gate criterion to: "Evaluator ran on a model distinct from all Phase-3 implementer/test-writer models for the wave (verify via dispatch record, not agent self-report — see #468). Cross-vendor family (e.g. GPT/Gemini) REQUIRED only when a multi-vendor router is configured; otherwise intra-vendor tier diversity + information-asymmetry wall is the documented fallback."
2. Same treatment for the LLM-preflight criteria in greenfield/brownfield/planning lobsters (`REQUIRED` → conditional on router config).
3. Record the actual evaluator model id in the evaluation report frontmatter so the gate is auditable.

## Environment / evidence

vsdd-factory 1.0.0-rc.21, greenfield on ArcavenAE/akey, Phase-4 dispatch 2026-07-04. No LiteLLM/multi-vendor router configured; holdout-evaluator can only be a Claude model.

## Related

- #410, #417 (platform-inherited model routing family)
- #468 (dispatched-model fidelity / self-report unreliability — the verification mechanism this gate needs)
- #260 (gate self-ratification pressure)

Suggested labels: `bug`, `workflows`, `phase-4`, `gates`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.