BOHICA-LABS / BOHICA-LABS/vsdd-factory

enhancement(scaffold-claude-md/health-check): no check that CLAUDE.md's documented build/test commands are runnable & non-vacuous — wrong path/missing -ginclude_subdirs runs 0 tests, exits green

Open
#331 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
1
Avg merge
6h 43m
Merged PRs (30d)
29

Description

## Class of failure

Nothing validates that the **build/test commands documented in a project's `CLAUDE.md` (or equivalent agent-facing doc) are actually runnable and non-vacuous**. A documented test command can match **zero files** and still "succeed" (exit 0, "0 tests run"), so every agent that follows the doc gets a green that proves nothing — a latent false-green vector seeded at the documentation layer.

## Pattern observed (ftc-blue pilot)

The project `CLAUDE.md` prescribes:

```
GUT unit tests: godot --headless --script addons/gut/gut_cmdln.gd -gtest=res://tests/unit/ -gexit
Lint GDScript: gdlint scripts/
```

Three independent defects, all silently green:
1. **`res://tests/unit/` does not exist** — the real path is `test/unit/` (singular). The glob matches nothing → GUT runs **0 tests** and exits cleanly. Any agent trusting this command "passes" while testing nothing.
2. **`-gtest=` does not recurse** — GUT needs `-gdir … -ginclude_subdirs` to walk subdirectories; without it, nested suites are skipped even if the path were correct.
3. **`gdlint scripts/`** omits `test/` and `autoloads/` — lint coverage silently excludes two of the three source roots, so style/structure violations there never surface.

The correct, verified commands for this project are:
```
godot --headless -s addons/gut/gut_cmdln.gd -gdir=res://test/ -ginclude_subdirs -gprefix=test_ -gsuffix=.gd -gexit
gdlint scripts/ test/ autoloads/
```

## Provenance note (scope)

In this pilot the bad `CLAUDE.md` was hand-authored (not emitted by `scaffold-claude-md`), so this is **not** a claim that the engine *generates* a broken command. The engine-level gap is the **absence of any check** that the commands a `CLAUDE.md` documents are runnable and non-vacuous — and `scaffold-claude-md` is well-positioned to *introduce* such a check for the docs it does generate. Relatedly, #323 (demo-recorder uses `gut_cli.gd` not `gut_cmdln.gd`) is the same family of "documented GUT invocation is wrong," but scoped to a different agent and a runtime-loadability bug rather than a vacuous-glob bug.

## Why existing issues don't cover it

- **#174** (CLAUDE.md health-check + threshold-driven compaction): scoped to *size* governance, not command validity.
- **#323**: demo-recorder's GUT entry point, not the project doc's test/lint command set, and not the "matches zero files → vacuous green" class.
- **#295 / #302**: CLAUDE.md *content placeholders* and *git-rule scoping*, not command runnability.

## Proposed remediation

A `CLAUDE.md` (or build-doc) health check / `scaffold-claude-md` post-step that, for each documented build/lint/test command:
1. Confirms referenced paths exist (`res://tests/unit/` vs `test/unit/`).
2. Runs the command in a dry/collect mode and **fails if it collects zero tests / zero files** (vacuous-green guard — "documented test command runs N>0 tests").
3. For GUT specifically, warns when a directory selector lacks `-ginclude_subdirs` (recursion footgun) and when lint roots omit known source directories.

Moves "documented command silently tests nothing" from "discovered by accident months later" to "caught when the doc is written or health-checked."

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.