BOHICA-LABS / BOHICA-LABS/vsdd-factory

enhancement(orchestrator): add smoke sentinel gate to per-story-delivery Step (c3) and Wave Integration Gate — quality-gate-green ≠ operator-safe-green

Open
#487 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Defect class

Every VSDD-managed project that ships a runnable binary or service has a class of regression the current per-story delivery loop cannot catch:

| Gate | Verifies | Misses |
|---|---|---|
| test-writer / red-gate | tests exist + fail before impl | anything the tests don't happen to cover |
| implementer TDD | unit tests pass | operator-visible behavior at the binary boundary |
| adversary convergence (BC-5.39.001) | diff review, 3× clean streak | can't execute the binary |
| demo-recorder (Step d) | captures whatever the binary does | records exit 1 on `--help` faithfully if that's what it does |
| pr-manager 9-step | CI green, review approved | CI runs the same unit suite; operator boundary uncovered |
| Wave Integration Gate Step a | full test suite on merged develop | same coverage gap at wave-scale |

`per-story-delivery.md` (plugin `1.0.0-rc.21`) treats `just test` green + adversary green + demo recorded as sufficient for green. It is not. Between the implementer's `git add` and the pr-manager's `gh pr merge`, nothing in the loop executes the artifact as an operator would.

## Observed failure (switchboard-blue, 2026-07-04)

In one afternoon, a tutorial-walk on `docs/getting-started.md` (post-SHIPPED steady-state operator-doc audit) caught **four operator-boundary regressions** — S1/S3/O1/O3 in `.factory/STATE.md` drift register:

- **S1** — `switchboard --help` printed a diagnostic to stderr and exited **1**. BC-2.07.002 EC-003 Ruling A requires stdout + exit 0.
- **S3** — version banner hardcoded to a literal instead of ldflags-injected from `main.version`.
- **O1** — `sbctl --version` flag was **missing entirely**.
- **O3** — sbctl-a Homebrew formula packaged without version-ldflags wiring; every packaged binary reported `sbctl dev`.

Each had shipped through the factory's Quality Gate (`just fmt` + `just lint` + `just test` all green). Each was a "one-line" change that looked mechanical in code review. Every one was reachable only by executing the entry-point binary as an operator would — none were caught by the unit suite, none by the adversary reviewing the diff, none by the demo-recorder (which captured the broken behavior faithfully and passed the recording along).

This is not a switchboard-specific quirk. The deliverable is a binary; every VSDD-managed project shipping an operator binary or a running service has the same shape.

## Remediation prior art (same session, switchboard-blue)

Shipped locally as three PRs (all merged, all green):

- **PR [#77](https://github.com/ArcavenAE/switchboard/pull/77)** — fixed the four regressions (BC-2.07.002 EC-003 Ruling A `--help` stdout + exit 0, ldflags-driven version banner, `sbctl --version` flag, sbctl-a ldflags wiring).
- **PR [#78](https://github.com/ArcavenAE/switchboard/pull/78) (Plan A)** — added `test/smoke/invariants.sh` (INV-1..INV-8 sentinel harness, ≤5s wall-clock, JSONL report on failure), `just smoke-quick` recipe, `Sentinel invariants` CI step, and `docs/architecture.md §Smoke invariants` catalog with an "add-a-new-invariant" playbook.
- **PR [#79](https://github.com/ArcavenAE/switchboard/pull/79) (Plan C)** — `## Blast Radius` PR-template block with a CI check that regex-validates the three prompts (operator-visible surfaces / silent-failure risk / smoke gate touched), CONTRIBUTING.md §Blast radius rationale with good/bad examples.

Together: automated behavioral gate + human-review declaration gate. The class of regression that produced this issue is now guarded in switchboard-blue.

**BUT:** the gate fires only because the human orchestrator remembered to add it. Nothing in `agents/orchestrator/per-story-delivery.md` tells any orchestrator running any other project to invoke it. That is the same failure mode that let the regressions ship in the first place — an existing gate that the orchestrator does not run before declaring green. This issue moves the gate from project-specific to platform-wide.

## Proposed fix

### Part 1 — new Step (c3) in `agents/orchestrator/per-story-delivery.md` §Per-Story Delivery

Insert between Step (c2) adversary convergence and Step (d) demo-recorder:

```
c3. Spawn implementer: "Run smoke sentinel gate on .worktrees/STORY-NNN/.
Command: `just smoke-quick` (or project-equivalent behavioral-surface
sentinel harness — reference `docs/architecture.md §Smoke invariants`
for the sentinel catalog). Exit 0 = pass. Exit 1 = regression;
route back to implementer with the sentinel report as fix input.
Exit 2 = harness broken; escalate to human.
Step (d) MUST NOT begin while smoke gate is failing."
```

Rationale: the smoke gate runs against the built binary — the same artifact the operator executes. Sentinels assert behavioral surfaces (exit code, stream direction, banner content, `--help` reachability without socket-open, etc.) that unit tests cannot cover without heroic scaffolding.

### Part 2 — new Step (aa) in Wave Integration Gate

Insert between Step (a) `implementer: "Run FULL test suite on merged develop"` and Step (b) `adversary: "Review combined wave diff..."`:

```
aa. Spawn implementer: "Run smoke sentinel gate on merged develop tip.
Command: `just smoke-quick`. Same exit-code convention as per-story c3.
On failure: fail the wave gate immediately; do NOT proceed to adversary
review of a wave that ships an operator-boundary regression."
```

Rationale: even if every story's per-story gate passed, a wave-level merge conflict or feature interaction can re-open a regression. Cheap to run (≤5s in switchboard-blue); catches inter-story interference at the earliest wave gate that has a built binary.

### Part 3 — auto-scaffold in `phase-1-cicd-setup`

During CI/CD setup, if the project ships a runnable binary or service (heuristic: `Cargo.toml [[bin]]` / `go.mod` with `cmd/` / `package.json bin` / `Dockerfile CMD/ENTRYPOINT`), scaffold:

- `test/smoke/invariants.sh` (behavioral-surface sentinels tailored to detected entry points)
- `just smoke-quick` recipe
- CI `Sentinel invariants` step in the Quality Gate workflow
- `docs/architecture.md §Smoke invariants` section with catalog table + "Adding a new invariant" playbook

Blueprint files to mirror: [`test/smoke/invariants.sh`](https://github.com/ArcavenAE/switchboard/blob/develop/test/smoke/invariants.sh), [`.github/workflows/ci.yml` Sentinel invariants step](https://github.com/ArcavenAE/switchboard/blob/develop/.github/workflows/ci.yml), [`docs/architecture.md §Smoke invariants`](https://github.com/ArcavenAE/switchboard/blob/develop/docs/architecture.md#smoke-invariants).

### Part 4 — contract rules for the sentinel harness

Mandatory, otherwise the gate becomes review theatre:

1. **Behavioral only.** Every assertion checks exit code, stream direction (stdout vs stderr), or substring presence. Cosmetic diffs (exact whitespace, exact ordering, colour codes, timestamp format) are forbidden. Reviewers reject cosmetic sentinels.
2. **Paired docs.** Every new invariant requires a paired update to `docs/architecture.md §Smoke invariants` documenting the defect class it guards against. An invariant without a documented rationale is a phantom assertion.
3. **Fail loud, fail fast.** Failure blocks merge. Harness uploads its JSONL report as a CI artifact on failure for post-mortem.
4. **Isolated.** Runs in a fresh tmpdir with `trap` cleanup. Does not touch user state.
5. **Three-state exit codes.** `0` = all-pass. `1` = regression (one or more invariants failed). `2` = harness broken (binary missing, tmpdir unwritable). CI distinguishes "smoke found a bug" from "smoke can't run."
6. **Each invariant traces back to a concrete regression it prevents.** No speculative sentinels.

## Acceptance criteria

- [ ] `agents/orchestrator/per-story-delivery.md` §Per-Story Delivery has a Step (c3) that mandates smoke-gate green before Step (d).
- [ ] `agents/orchestrator/per-story-delivery.md` §Wave Integration Gate has a smoke-gate step (aa) that fails the wave on regression.
- [ ] `phase-1-cicd-setup` auto-scaffolds the harness for binary-shipping projects with the contract rules documented.
- [ ] `docs/FACTORY.md` or `docs/VSDD.md` picks up "smoke sentinel" alongside adversary-convergence + red-gate as a first-class quality gate.
- [ ] `deliver-story` skill updated to match `per-story-delivery.md` (canonical-source rule).

## Related — distinct from

- **#177** (hollow-demo / false-confidence checker) — asks "does this test assert substance?" C6 asks "does the binary do the right thing at the operator boundary?" Same failure category (green-but-wrong), different lens. Complementary: #177 is a review pass; C6 is an execution gate.
- **#433** (dynamic-launch attestation for visual/UI stories) — scoped to renderable surfaces with multimodal review. C6 is scoped to CLI/service binaries with deterministic behavioral assertions. Same class ("gates verify only static properties"); different surface and enforcement mechanism. Together they close the class across all deliverable shapes.
- **red-gate** (BC-5.38.001) — asserts tests fail before implementation. C6 asserts operator surfaces stay green after implementation. Bookends of TDD.
- **adversary convergence** (BC-5.39.001) — reviews the diff. C6 exercises the binary.
- **#472** (test-writer scope) — test-writer produces failing unit tests from BCs; smoke sentinels are behavioral-surface assertions against the built artifact. Distinct scope.

## Priority

**HIGH** — every VSDD-managed project shipping an operator binary or service has this failure mode. switchboard-blue hit four instances in one afternoon during steady-state audit; each shipped through the full pipeline unnoticed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.