BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(story-writer+test-writer): AC-named test targets must be verified instantiable in the test framework; silent test-type substitution must escalate, not comment

Open
#481 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Summary

Two coupled process failures observed when an AC named a behavioral test against a class that cannot be instantiated inside the test framework:

1. **AC authoring gap:** the AC specified a behavioral GUT test against a class that extends `SceneTree` (which cannot be instantiated inside GUT's test runner). The test-writer discovered the incompatibility but silently substituted source-scan tests and noted the gap only in a test-file comment — no escalation, no known-issue record, no deviation flag.

2. **Silent substitution propagates unchecked:** the substitution (behavioral test → source-scan) was not visible to any gate between test-writing and convergence. The gap was found only three review passes later by an adversarial reviewer reading the test file.

## Observed pattern (private Godot 4.7 project; identifiers scrubbed)

An AC read: *"GUT behavioral test: given [state], assert [observable behavior]."* The target class extended `SceneTree` — a non-instantiable base class under GUT. The test-writer wrote source-scan tests instead (grep-style structural assertions) and added a comment in the test file noting the limitation. No deviation flag, no story annotation, no escalation to the story-writer or orchestrator. The test file passed the Green Gate (source-scan tests pass trivially). Three adversarial passes later, the reviewer found the behavioral test absent and escalated.

## Root cause

**AC authoring:** the AC named a specific test type and target without verifying the target is reachable by that test type in the project's test framework. The incompatibility was latent in the spec from authoring time.

**Silent substitution:** no protocol required the test-writer to escalate a mandated-test incompatibility. The test-writer's path of least resistance was a passing substitute that fulfilled the structural form of an AC ("test exists") without fulfilling its behavioral intent.

## Proposed remedy

### For AC authoring (story-writer checklist item)

When an AC names a test (GUT behavioral test, integration test, etc.) against a specific class or component:
1. Verify the target class is instantiable (or otherwise reachable) in the test framework at the project's pinned version.
2. If the target is NOT instantiable, the AC must either: (a) name the extractable helper/inner function or a testable adapter that IS reachable, or (b) explicitly document scan-only coverage as a known limitation with a KI record.
3. Adversarial spec review checklist item: "For every AC that names a test target: is that target instantiable in the test framework?"

### For test authorship (escalation contract)

When a test-writer cannot implement a mandated test type as specified (instantiation failure, missing seam, framework incompatibility):
1. **This is an escalation, not a comment.** The test-writer must flag the incompatibility to the orchestrator or in the story's KI register before shipping the test suite.
2. Source-scan or structural tests substituted for mandated behavioral tests must be labeled explicitly in the AC coverage manifest as `structural` (not `behavioral`), and the deviation must be recorded as a KI.
3. A test-file comment is NOT a substitute for a KI record or an escalation — it is invisible to all gates downstream of the test-writer.

## Related

- #353 (coverage-manifest gate with per-AC/per-EC behavioral|structural tagging — the silent substitution would be surfaced if the manifest required explicit tagging of each AC's test type)
- #475 (stub-architect test-file authorship — a different variant of a stage-gate agent substituting its own judgment for the specified behavior without escalating)
- #150 (story clause contradicts mandatory pipeline mandate — the story-writer authoring an AC that mandates an unreachable test is a similar "spec says X but pipeline cannot do X" class)
- #327 (AC trace citations not resolved — same family: AC claims behavioral coverage that the cited artifact cannot provide)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.