BOHICA-LABS / BOHICA-LABS/vsdd-factory
process-gap(test-harness): Godot stale class-cache produces phantom failure cascades misdiagnosed as cross-story regression
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Observation
For Godot products, `godot --headless -s addons/gut/gut_cmdln.gd …`
relies on the **engine-resolved global class cache** (built lazily when
Godot opens the editor or runs a full export). On a fresh clone, after
dependency churn, or after a branch switch that adds/renames
`class_name`-typed scripts and `.import` siblings, that cache can be
stale or missing.
In stale-cache mode, GUT runs to completion **with exit 1 and a large
number of failures** that have nothing to do with test logic. The cause
is uniform but the failure messages are scattered:
- `Invalid call. Nonexistent function 'new' in base 'GDScript'.` (when
preloads via `class_name` resolve to Nil)
- `'X' on a base object of type 'Nil'` (autoload `class_name` resolution)
- `Cannot make signal assertions because the object X is not being
watched` (autoload itself was Nil at watch-time)
- `[BC-X] resource not found … — skipping` (`.import` siblings not yet
registered)
In a recent session this surfaced as **a multi-hundred-failure cascade**
on the main branch, which the orchestrator initially routed as a
cross-story regression hotfix. A triage agent re-ran the **identical
GUT command** after a one-time `godot --headless --editor --quit` (a
no-op editor open that rebuilds the class cache) and obtained
**0 failures / full suite passing / exit 0** on the same commit, same
hardware, same engine version.
## Cost to the pipeline
A false-positive failure cascade was almost spawned as a hotfix-line
investigation; would have wasted a session before being correctly
diagnosed as harness state, not product state. The triage agent caught
it only because she ran the full suite from two different worktrees
with different cache states.
## Proposed framework remediation
For Godot products:
1. **canonical-test-command preflight**: orchestrator-dispatched GUT
invocations should prepend `godot --headless --editor --quit` (one-shot
cache rebuild, ~3 s wall) **on a freshly-checked-out worktree or after
any branch switch that touches `class_name`/`.import`-affected files**.
Track via a `.factory/cache-rebuild-token` checkpoint per worktree.
2. **harness-vs-product-failure classifier**: the test-runner skill (or a
wrapper) should detect the signature of cache-stale failures (large
failure count with the diagnostic strings above, no test-logic
stacktraces) and emit a `HARNESS_STATE_SUSPECTED — rebuild class cache
and retry` sentinel BEFORE the orchestrator routes findings.
3. **CI gate**: the project's CI workflow should warm the cache once per
runner, not assume warm cache. Some CI runners reuse Godot caches;
some don't. Add a `mise task` or `Makefile` target encoding the
warmup.
## Distinct from existing upstreams
- Distinct from #294 (baseline-vs-main verification): #294 protects
against "implementer claims pre-existing failures" — this protects
against "everyone agrees there ARE failures" but the cause is harness.
- Distinct from #259 (code-only convergence): the failures here aren't
even in the code; they're in the engine state.
- Distinct from #309 (ground-truth capture lint-class invisible): that's
about CI-gate visibility; this is about test-runner state itself.
## Severity
MEDIUM. Loss measured in session triage cost, not ship blocker —
ultimately diagnosable. But the misroute risk is real: a session that
believes a large number of tests are failing will spawn fix-stories
that turn out to be no-ops. One occurrence per pipeline costs 1-2 hours
of agent time.
## Evidence
Same SHA, same hardware: pre-`--editor --quit` produces a large failure
cascade; post produces 0 failures. Engine: Godot 4.x; GUT: latest 9.x.
Anonymized story-id available on request; reproduction signature is
engine-generic.
Contributor guide
Assessment
This issue has not been assessed yet.