BOHICA-LABS / BOHICA-LABS/vsdd-factory

process-gap(test-writer): GUT assert_push_error is CONSUMING — documented error side effects are fixture contract, not noise; tests not consuming them fail post-implementation

Open
#499 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Pattern (Godot/GUT-specific test-authoring trap)

GUT's `assert_push_error` (and its variants) is a **consuming** assertion: it expects exactly one `push_error` / `push_warning` call to be triggered by the expression it wraps, and it consumes that error from the error queue. If the code under test is specified to emit a documented error side effect (e.g., "logs an error when called in an invalid state"), and the test does **not** call `assert_push_error` to consume it, GUT's post-test error-queue check fires and the test fails as "Unexpected Errors."

The trap: test-writers author tests for the primary behavioral assertion (return value, state change, signal emitted) but treat the documented error side effect as an implementation detail to be ignored. The tests pass at Red Gate (stubs emit nothing). They fail at Green Gate or post-implementation because the implemented code now correctly emits the documented error — and nothing is consuming it.

## Observed recurrence

This trap was hit twice in a single delivery wave (two independent stories, two independent test-writers, same session). In both cases:

1. The BC documented the error-emission behavior explicitly as a side effect or edge-case postcondition.
2. The test-writer wrote tests for the primary assertion but did not wrap the triggering call in `assert_push_error`.
3. Tests passed Red Gate (stub bodies emit nothing).
4. After implementation, the same tests failed with "Unexpected Errors" — the implementation correctly emitted the error as specified, but the test had no consumer.

The fix in both cases was to add `assert_push_error(...)` wrapping the call that triggers the documented error. This is not an optional assertion — it is part of the test fixture's contract with GUT's error-queue mechanism.

## Root cause

The GUT documentation describes `assert_push_error` as an assertion that verifies an error is pushed. Test-writers read it as "use this when I want to *test* the error path." The consuming semantics — that calling the expression inside `assert_push_error` is the only safe way to exercise code that emits a documented error — are not emphasized. The consequence is that tests can be authored that appear complete (they test the behavior they were written for) but will fail once the implementation is correct.

## Framework ask

1. **Test-writer agent instruction (GUT projects):** when the BC specifies any error/warning emission as a side effect, postcondition, or edge-case behavior, the test file MUST include `assert_push_error` (or `assert_push_warning`) wrapping every call that triggers the documented emission. An unflagged `push_error`/`push_warning` call in the implementation will cause GUT's post-test check to fire even if the test's primary assertion passed.

2. **Story-writer BC guidance:** for any AC or postcondition that documents an error/warning side effect, add a note: "test must consume this error emission with `assert_push_error`." This converts the side-effect documentation into an explicit fixture-contract requirement.

3. **Red-Gate validation:** error-emitting behaviors should have a Red Gate test that not only asserts the primary behavior but also includes the `assert_push_error` consumer — ensuring the consumer is in place before the implementation emits.

## Why this is a recurring authoring trap (not a one-off)

Two instances in one wave, two different test-writers, both following the same pattern. The consuming semantics of `assert_push_error` are non-obvious: it looks like a verification tool, not a queue-management primitive. Any test-writer who has not already hit this failure will author the same mistake. The framework's test-writer agent definition or GUT-specific guidance should make the consuming semantics explicit.

## Related upstream issues

- #481 (AC-named test targets must be verified instantiable — test-writer escalation contract for incompatible test types; same "test-writer hits a framework semantic trap" class)
- #475 (stub-architect test-file authorship at stub stage — Red Gate contract; this issue is about the Green Gate consequence of incomplete fixture contracts)
- #353 (vacuous-pass ratio at Red Gate — a related Red-Gate quality gap; this is a post-impl failure, not a Red Gate quality issue)

*(Framework-pattern detail only. Godot 4.x / GUT test harness. From a private project running the factory.)*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.