bench-cli tests time out under load — every case spawns a fresh `npx tsx`
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 21m
- Merged PRs (30d)
- 41
Description
## Evidence
Found verifying #65's merge. A full `pnpm test` on `f6cd2ec` produced a fifth failure on top of #50's four:
```
× bench new > forwards the parent's model when the parent is on an auto router 30028ms
→ Test timed out in 30000ms.
✓ bench new > sends nothing when the parent is pinned to a model, so the role decides 13253ms
✓ bench new > sends nothing when the parent never said what it is 646ms
```
The same file immediately afterwards, in isolation:
```
✓ bench new > forwards the parent's model when the parent is on an auto router 771ms
Test Files 1 passed (1) Tests 10 passed (10)
```
771ms alone, over 30,000ms in the full suite. The test after it took 13,253ms against a normal 802ms — so this is not one bad case, it is the whole file degrading by one to two orders of magnitude under contention.
## Why
Every test in `tests/bench-cli.test.ts` shells out to a fresh process:
```ts
// tests/bench-cli.test.ts:80-84
async function runBench(env: Record, args: string[]) {
return exec("npx", ["tsx", join(process.cwd(), "src/cli/bench.ts"), ...args], {
env: { ...process.env, ...env },
});
}
```
Ten tests, ten `npx` resolutions and ten `tsx` cold starts, all competing with the rest of the parallel suite for the same cores. `testTimeout` is 30s (`vitest.config.ts`), which is generous for a 771ms operation and still not enough.
**This is not a regression from #65 and not the bug #65 fixed.** #65 was fixed-duration sleeps racing an async write in `registry.test.ts` and `rename.test.ts`; its own criterion — two concurrent `pnpm test` runs both passing — was verified on its branch and held. `bench-cli.test.ts` is untouched by that work and fails by a different mechanism: process spawn cost, not a sleep. It is filed separately for that reason.
The cost is the same one #65 was about: a green suite stops being evidence, and telling a real regression from a loaded machine takes two extra full runs.
## Acceptance criteria
- [ ] `tests/bench-cli.test.ts` does not pay a fresh `npx` resolution and `tsx` cold start per test
- [ ] The named cause is fixed rather than papered over with a longer `testTimeout` — raising the timeout would hide it, not solve it
- [ ] The file's ten cases still assert the same things; this is a harness change, not a coverage change
- [ ] The file passes in isolation and inside a full run on a loaded machine
- [ ] Two concurrent `pnpm test` runs both pass, still
- [ ] `pnpm typecheck` and `pnpm test` clean against the #50 baseline of 4 known failures
## Out of scope
- Raising `testTimeout` in `vitest.config.ts`. That is the workaround this ticket exists to avoid.
- The #50 four.
- `tests/registry.test.ts` and `tests/rename.test.ts` — done in #65, do not revisit.
- Any change to `src/cli/bench.ts` itself. The CLI is fine; how the test invokes it is not.
## Verification
```
pnpm typecheck
pnpm test
pnpm test & pnpm test; wait # both runs must pass
npx vitest run tests/bench-cli.test.ts # and report the per-test durations
```
## Related
- #65 (a different flake with the same symptom; merged at `f6cd2ec`)
- #71 (hard-coded `/tmp` homes; also found reviewing #65)
- #50 (the 4-failure baseline)
Contributor guide
Research direction
Start with tests/bench-cli.test.ts around runBench and check vitest.config.ts for the current timeout and execution setup. Run the named test file, then pnpm typecheck, pnpm test, and two concurrent pnpm test runs; done means the ten cases keep their assertions without per-test npx/tsx startup and pass under load.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100