BOHICA-LABS / BOHICA-LABS/vsdd-factory
process-gap(BC-5.38.001): Red Gate density count over-reports when stubs are behind build tags / feature flags / test markers — publish red_default + red_gated
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Summary
The BC-5.38.001 Red Gate is measured by counting failing tests emitted by a story's failing-test batch and asserting they exceed a density threshold. When a story's file structure requires tests behind a build tag (e.g. Go's `//go:build integration`), those tests are excluded from the default `go test ./...` (or `cargo test`, `pytest -q`) run — meaning:
**The Red Gate density count over-reports what the test-writer's default runner sees vs. what the tree actually contains.**
Observed cleanly in `switchboard-blue` S-6.05 stub burst (2026-07-01, plugin `1.0.0-rc.21`).
## Repro
Story S-6.05 file structure:
- `internal/svtnmgmt/svtnmgmt_test.go` — 4 unit-test stubs
- `cmd/sbctl/admin_test.go` — 4 CLI-test stubs
- `cmd/switchboard/admin_handlers_e2e_test.go` — 2 stubs **behind `//go:build integration`** (VP-048 E2E)
Total stubs on disk: 10 (8 + 2 integration). Red Gate density counts 10.
Default `go test ./...` in the worktree runs 8 stubs; the 2 integration stubs are excluded. `go test -tags=integration ./cmd/switchboard/...` runs all 10 — but no factory-standard command invokes this by default.
## Why the current protocol is silent
BC-5.38.001 is written test-runner-neutral. It says: "stubs compile, all new tests must be red." It does not distinguish:
- Tests that run in the default build.
- Tests that require a build tag (Go), a feature flag (Rust `cfg(feature = "…")`), a marker (pytest `@pytest.mark.integration`), or an env var (`RUN_INTEGRATION=1`).
- Tests behind an ignored/skip attribute (`#[ignore]`, `t.Skip`, `unittest.skip`).
The stub-architect and test-writer both correctly place tests where the story spec directs, but nothing in the protocol says the Red Gate check must:
1. Verify each stub is discoverable by the runner it will be counted against.
2. OR, publish two counts: `red_default` (default build) and `red_all` (with all gates flipped on).
3. OR, mandate that a story's stubs may not fall behind build tags without a matching CI job that runs them.
Failure modes observed / possible:
- **Red Gate count inflation:** density = 10, but only 8 discoverable → 2 stubs are "structurally present but not part of the assertion the number is proving."
- **False Red Gate pass:** if a full-conversion story places ALL its stubs behind a build tag with no CI job flipping the flag, the Red Gate literally proves nothing about the default build.
- **Silent green-drift under implementer:** the implementer sees `go test ./...` pass and thinks the story is delivered, but the 2 integration stubs are still `t.Fatal("todo: ...")`. Adversary might catch it if it runs `go test -tags=integration`; not guaranteed.
## Adjacent-but-distinct
- #330 (headless test blind to render modality) — modality mismatch at the runner level (headless vs. rendered), this is exclusion at the *build* level (tag flipped off entirely).
- #354 (BC-mandated seams silently omitted) — omission at the seam level, this is omission at the runner-discovery level.
- #360 (panic-sourced red → vacuous green) — vacuous-green after implementation; this is *pre-implementation* count inflation.
None of the above name the "runner-discoverability vs. tree-presence" gap.
## Proposed fix
Two-part:
### Part 1 — Red Gate check must publish both counts
Extend BC-5.38.001 (or emit a supplement, e.g. `BC-5.38.006`) requiring the stub commit report and the Red Gate verification step to publish:
```
Red Gate — stub coverage
red_default: N tests failing under default build (e.g. `go test ./...`)
red_gated: M tests failing under all gates flipped on
(e.g. `go test -tags=integration,e2e ./...`)
gated_stubs: list of files/tests behind gates
- cmd/switchboard/admin_handlers_e2e_test.go (build tag: integration)
```
Density threshold applies to `red_default`. `red_gated` must be non-empty when the story spec assigns AC coverage to a gated file — the adversary can then check the gated tests are ALSO red before green-claim.
### Part 2 — Story-spec + CI coupling requirement
If a story spec's file structure places tests behind a build tag / feature flag / marker:
- Story spec must name the CI job (or `just` recipe / `justfile` target) that will run those tests, or state that no CI job runs them today (a drift item).
- If no CI job runs them, the wave adversary must surface a first-class finding: "Story S-N.MM has M gated stubs with no CI job that flips the gate — Red Gate assertion applies to a tree the runner never sees."
- Story checklist gains: `[ ] All gated tests have a named runner (CI job / just recipe / test tag command)`.
### Part 3 (optional) — factory-standard runner discovery
Emit a factory-wide convention that stories with integration/e2e tests must include a `just test-` recipe (or equivalent) invoked as part of the wave-gate CI, so gated tests are guaranteed to run at some point in the pipeline. Currently `switchboard-blue` has `just test-integration` for exactly this reason, but the pattern isn't ratified upstream.
## Materials
- Story: `switchboard-blue` `.factory/stories/S-6.05-svtn-destroy-lifecycle.md`
- Stub commit: forthcoming (branch `feat/S-6.05-svtn-destroy`)
- 8 default-runner stubs + 2 integration-tag stubs = 10 stubs on disk; default `go test ./...` sees 8.
## Severity
MEDIUM-HIGH. Silent Red Gate count inflation compounds over waves. Worst-case: a story with all-gated stubs and no CI job flipping the gate has a Red Gate assertion that literally proves nothing — but the density number is still published in the stub commit report and the story proceeds to green-claim.
## Cross-references
- Plugin version: `1.0.0-rc.21`
- Stack: Go 1.25.4 (build-tag exclusion); pattern is language-agnostic (Rust `cfg`, pytest markers, JUnit `@Tag`)
- Filing session context: `switchboard-blue` Wave-6 Tranche C, S-6.05 stub burst, 2026-07-01
- Related: #330, #354, #360, #418, #420 (companion process-gap filed same session)
---
**Filed by:** ArcavenAE/switchboard-blue orchestrator (Claude Code, `claude-sonnet-4-6`), Wave-6 Tranche C, 2026-07-01.
Contributor guide
Assessment
This issue has not been assessed yet.