ship: verify promised tests landed in diff before push
- Dominant language
- TypeScript
- Stars
- 133k
- Forks
- 19.9k
- Avg merge
- 18h 46m
- Merged PRs (30d)
- 26
Description
### Problem
`/plan-eng-review` has a Test review section (`plan-eng-review/SKILL.md.tmpl:153-159`) that produces a codepath diagram and asks the user to add tests to the plan before implementation. Those tests end up as plan items.
`/ship` has a Plan Completion Audit (`ship/SKILL.md:1565-1699`) that extracts plan items, categorizes them as CODE / TEST / MIGRATION / CONFIG / DOCS (`ship/SKILL.md:1625-1627`), and reports DONE / PARTIAL / NOT DONE / CHANGED. Results write to JSONL at `ship/SKILL.md:2846` as `plan_items_total` / `plan_items_done`.
Step 8 verifies "was the plan done." It does not verify "were the promised tests delivered." TEST-category items aren't specially tracked. Nothing cross-references the diff for evidence that a test file exists for each promised test. The coverage gate at `ship/SKILL.md:1505-1518` checks aggregate coverage, which doesn't catch promise-specific gaps. You can hit the coverage threshold with tests for other things and still have shipped an untested feature.
So a disciplined `/plan-eng-review` ... `feat: implement X` ... `/ship` loop can silently drop the test at step 2. Reviewer asked for it. Plan listed it. Implementer didn't write it. Ship didn't notice.
### Evidence
Ran `/retro global 7d` across my 8 active repos (2026-04-11 through 2026-04-18):
- 172 commits
- 67 `feat:` (34%), 56 `fix:` (28%)
- **6 `test:` commits (3%)**
Some tests landed inside `feat:` commits. MSBP v2's Playoff War Room shipped 66 tests in one feature commit, that's the exception. The other 4 feature-heavy repos ran 10-25% feat-commits-with-sibling-test-files by inspection, well below what the plans promised.
Every plan was reviewed by `/plan-eng-review`. The review listed specific tests. They just didn't all land.
Not unique to my repos. Anyone following the full review-implement-ship flow is exposed to it.
### Proposed fix
Sub-step in `/ship` Step 8. Pull the test-related plan items, check the diff, print what's missing.
I could do this small, medium, or big. I'd start small so real users see the numbers before we add friction.
**Small (informational only, ~150 lines in `ship/SKILL.md.tmpl`):**
- Pull plan items that are test-related (category tag, or the text mentions test/verify/cover/assert)
- Scan the diff for evidence (new test files, new test function signatures per language)
- Print in ship output: "Plan promised N tests. M landed. K missing: [list]"
- Add `test_items_promised` and `test_items_verified` to the existing JSONL. No gate.
**Medium (soft gate, +50 lines):**
If K missing > 0, AskUserQuestion, same shape as the coverage gate at `ship/SKILL.md:1505-1518`:
- A) Add the missing tests now
- B) Ship anyway, they were intentionally dropped (notes it in PR body)
- C) Ship anyway, follow up in a P1
**Big (not this PR):**
Formalize a `## Tests` heading in plan files. `/plan-eng-review` writes promises to a JSONL, `/ship` reads it for exact cross-reference. Touches two skills and the plan format, so better as a follow-up if the small version earns it.
### Questions before I fork
1. Small first, or go straight to the soft gate? My gut says small.
2. Keyword heuristic for test items, or formalize a `## Tests` heading in plans? Heuristic is less invasive, happy either way.
3. Where should the per-language test-function patterns live? New file in `bin/`, or inline in the SKILL.md.tmpl?
4. Am I missing an existing effort? I grepped `test promise`, `test gate`, `plan completion`, `tests promised` and got nothing, but figured I'd ask.
If this conflicts with anything you've got in flight, say so. I can put up a draft PR at the small scope in a couple of days if you want to see code.
Thanks for gstack, it runs every PR I ship.
### References
- `plan-eng-review/SKILL.md.tmpl:153-159` Test review section
- `scripts/resolvers/testing.ts:182-192` TEST_COVERAGE_AUDIT_PLAN resolver
- `ship/SKILL.md:1565-1699` Plan Completion Audit (Step 8)
- `ship/SKILL.md:1625-1627` category taxonomy
- `ship/SKILL.md:1505-1518` coverage gate (soft-gate precedent)
- `ship/SKILL.md:2846` JSONL metrics write
- `bin/gstack-review-log` artifact-passing pattern
Contributor guide
Assessment
This issue has not been assessed yet.