jobizzness / jobizzness/bench

bench-cli tests time out under load — every case spawns a fresh `npx tsx`

Open Beginner friendly
#72 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
1h 21m
Merged PRs (30d)
41

Description

## Evidence

Found verifying #65's merge. A full `pnpm test` on `f6cd2ec` produced a fifth failure on top of #50's four:

```
× bench new > forwards the parent's model when the parent is on an auto router 30028ms
→ Test timed out in 30000ms.
✓ bench new > sends nothing when the parent is pinned to a model, so the role decides 13253ms
✓ bench new > sends nothing when the parent never said what it is 646ms
```

The same file immediately afterwards, in isolation:

```
✓ bench new > forwards the parent's model when the parent is on an auto router 771ms
Test Files 1 passed (1) Tests 10 passed (10)
```

771ms alone, over 30,000ms in the full suite. The test after it took 13,253ms against a normal 802ms — so this is not one bad case, it is the whole file degrading by one to two orders of magnitude under contention.

## Why

Every test in `tests/bench-cli.test.ts` shells out to a fresh process:

```ts
// tests/bench-cli.test.ts:80-84
async function runBench(env: Record, args: string[]) {
return exec("npx", ["tsx", join(process.cwd(), "src/cli/bench.ts"), ...args], {
env: { ...process.env, ...env },
});
}
```

Ten tests, ten `npx` resolutions and ten `tsx` cold starts, all competing with the rest of the parallel suite for the same cores. `testTimeout` is 30s (`vitest.config.ts`), which is generous for a 771ms operation and still not enough.

**This is not a regression from #65 and not the bug #65 fixed.** #65 was fixed-duration sleeps racing an async write in `registry.test.ts` and `rename.test.ts`; its own criterion — two concurrent `pnpm test` runs both passing — was verified on its branch and held. `bench-cli.test.ts` is untouched by that work and fails by a different mechanism: process spawn cost, not a sleep. It is filed separately for that reason.

The cost is the same one #65 was about: a green suite stops being evidence, and telling a real regression from a loaded machine takes two extra full runs.

## Acceptance criteria

- [ ] `tests/bench-cli.test.ts` does not pay a fresh `npx` resolution and `tsx` cold start per test
- [ ] The named cause is fixed rather than papered over with a longer `testTimeout` — raising the timeout would hide it, not solve it
- [ ] The file's ten cases still assert the same things; this is a harness change, not a coverage change
- [ ] The file passes in isolation and inside a full run on a loaded machine
- [ ] Two concurrent `pnpm test` runs both pass, still
- [ ] `pnpm typecheck` and `pnpm test` clean against the #50 baseline of 4 known failures

## Out of scope

- Raising `testTimeout` in `vitest.config.ts`. That is the workaround this ticket exists to avoid.
- The #50 four.
- `tests/registry.test.ts` and `tests/rename.test.ts` — done in #65, do not revisit.
- Any change to `src/cli/bench.ts` itself. The CLI is fine; how the test invokes it is not.

## Verification

```
pnpm typecheck
pnpm test
pnpm test & pnpm test; wait # both runs must pass
npx vitest run tests/bench-cli.test.ts # and report the per-test durations
```

## Related

- #65 (a different flake with the same symptom; merged at `f6cd2ec`)
- #71 (hard-coded `/tmp` homes; also found reviewing #65)
- #50 (the 4-failure baseline)

Contributor guide

Open the contributing guide

Research direction

Start with tests/bench-cli.test.ts around runBench and check vitest.config.ts for the current timeout and execution setup. Run the named test file, then pnpm typecheck, pnpm test, and two concurrent pnpm test runs; done means the ten cases keep their assertions without per-test npx/tsx startup and pass under load.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.