garrytan / garrytan/gstack

/ship Step 5 hardcodes `bin/test-lane` + `npm run test` regardless of detected framework

Open
#1,069 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
133k
Forks
19.9k
Avg merge
18h 46m
Merged PRs (30d)
26

Description

## What happened

Ran `/ship` in a Django-only project. Step 4's `{{TEST_BOOTSTRAP}}` correctly detected the runtime:

> Test framework detected: Python/Django with chores/tests.py. Skipping bootstrap.

But Step 5 then told the agent to run:

```bash
bin/test-lane 2>&1 | tee /tmp/ship_tests.txt &
npm run test 2>&1 | tee /tmp/ship_vitest.txt &
wait
```

Neither command exists in a Django project. Both are hardcoded — `bin/test-lane` is Rails-specific and `npm run test` is Node-specific. They appear to have leaked in from a Rails-with-Vitest project the skill was authored against.

## Where

`ship/SKILL.md.tmpl:148-160` — Step 5 runs both hardcoded commands unconditionally, without regard to what Step 4 just detected.

## Why it's a bug

This directly violates the project's own rule in `CLAUDE.md`:

> ## Platform-agnostic design
>
> Skills must NEVER hardcode framework-specific commands, file patterns, or directory structures. Instead:
> 1. Read CLAUDE.md for project-specific config (test commands, eval commands, etc.)
> 2. If missing, AskUserQuestion — let the user tell you or let gstack search the repo
> 3. Persist the answer to CLAUDE.md so we never have to ask again

Step 4 already follows this pattern (detects runtime, bootstraps if needed). Step 5 doesn't — it just assumes Rails + Vitest.

## Suggested fix shape

1. Read the test command from the project's `CLAUDE.md` (the `## Testing` section that the bootstrap flow writes). If present, run it.
2. If missing, `AskUserQuestion` and persist the answer back to `CLAUDE.md` so the next `/ship` doesn't re-ask.
3. Drop the hardcoded `bin/test-lane` + `npm run test` pair.
4. Guardrail: add a test that renders `ship/SKILL.md` and fails if it contains hardcoded framework commands outside a detection block, so the regression can't quietly come back.

## Repro

Run `/ship` in any non-Rails, non-Node project (Django, Go, Rust, etc). Step 5 will propose `bin/test-lane` + `npm run test`.

## Out of scope

Step 6 ("Eval Suites") also has Rails-specific path patterns and `bin/test-lane --eval` commands, but it self-skips cleanly on non-matching diffs ("No prompt-related files changed — skipping."), so it's cosmetic for now. Leaving that for a separate issue if it ever actually fires incorrectly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.