Select Desktop e2e specs by what a change can reach, instead of all 103
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 715
Description
The test planner sub-selects everywhere except at its single most expensive point. `Desktop e2e` is 103 tests treated as one indivisible unit, it takes 651s of a 1528s `test` job, and 57% of commits pay for it in full.
### Measurement
Replaying `planTests` against the last 300 first-parent commits:
| lane | commits | share |
|---|---|---|
| code | 283 | 94.3% |
| releaseContract | 216 | 72.0% |
| cliPackage | 206 | 68.7% |
| runtimeSandbox | 202 | 67.3% |
| **e2e** | **171** | **57.0%** |
| storybook | 155 | 51.7% |
| astryxSurface | 143 | 47.7% |
| stateRootCompat | 88 | 29.3% |
| appIcons | 65 | 21.7% |
| asfSource | 61 | 20.3% |
| full | 59 | 19.7% |
| storageStress | 17 | 5.7% |
Average workspaces selected: 5.29 of 7.
The narrow gates are well aimed — `storageStress` at 5.7%, `asfSource` at 20.3%, `appIcons` at 21.7% — and workspace tests are already sub-selected through the dependency closure. `e2e` is the exception: it is all-or-nothing.
Step timings from run 33511334930 (`test`, 25m28s):
| step | seconds | share |
|---|---|---|
| Desktop e2e | 651 | 43% |
| Run affected standard workspace tests | 146 | 10% |
| Run Runtime Host tests | 135 | 9% |
| Storybook smoke | 107 | 7% |
| Install dependencies + Build + Typecheck | 45 | 2.9% |
### What actually triggers it
Of the 112 non-full commits that selected `e2e`, by the first path that drove the selection:
```
58 apps/desktop/src/main/** 52%
23 apps/desktop/e2e/** the specs themselves
13 apps/desktop/src/renderer/**
9 apps/desktop/renderer-architecture.json
5 packages/ui/**
4 apps/desktop/electron-builder.config.mjs
```
The largest single driver is the Electron main process, and the 34 spec files it schedules assert renderer contracts — composer, sidebar, settings, session rail, onboarding, quote selection, prompt rail, module hub. Editing `app-update-service.ts` runs all 103 of them. The main process is genuinely exercised by a real-window launch, so this is not wrong; it is just far wider than the change warrants.
### Suggested direction
Derive which specs a change can reach, using the closure machinery #4461 added for the Windows recovery filter. `collectWorkspaceSourceClosure` already computes a transitive source closure from a set of entry points; running it over the 34 spec files gives a per-spec reachable set, and a changed file then selects only the specs whose closure contains it.
Same pattern, same tool, and the same known limit: the closure walks static imports, so it cannot see process boundaries — `fork()`ed fixtures and bundled worker entry points are invisible to it. The Windows lane pairs its derived filter with a nightly unfiltered run for exactly this reason, and this would need the same pairing rather than being trusted alone.
Two smaller things worth settling first:
1. **Measure before cutting.** Playwright currently runs with the dot reporter, so the log carries no per-test timings and the cold-start share of that 651s is unknown. Each test mkdtemps a userData dir and launches a real Electron window, waiting for the composer to mount; at 103 tests over 651s the average is 6.3s, and the distribution is flat (466s for the first 80, 180s for the last 23), which points at fixed overhead rather than a few slow cases. Adding a `line` reporter to `apps/desktop/e2e/playwright.config.ts` makes every subsequent run report this for free — better than spending a runner slot on a one-off dispatch.
2. **`workers: 1` and the sharding that does not exist.** The config sets one worker for a documented and sound reason: concurrent hidden windows throttle animation frames and share OS focus, invalidating geometry and focus contracts. The same comment says "CI shards run on isolated X displays, so jobs still overlap without sharing focus or a compositor" — but `shard` appears nowhere in `.github/workflows/`. Either that sharding was lost or it was never implemented, and the comment should stop describing it either way.
Note that sharding trades against what #4461 optimised for. Splitting 103 tests across 4 shards would take the step from ~10.8m to ~2.7m at the cost of three more concurrent runner slots, and #4461's whole argument is that a slot is scarcer than a minute on shared infrastructure. Selection reduces both; sharding trades one for the other. Prefer selection.
### Also worth a look
`full` fires on 19.7% of commits. `FULL_SUITE_FILES` includes `package-lock.json`, so every dependency bump runs every surface. Whether that is the intended cost is worth confirming separately.
### Not in scope
Task-level output caching (Turborepo, Nx). It was considered and does not apply: `npm ci` is 17s, `Build` 19s, `Typecheck` 9s — 2.9% of the job combined, already covered by `setup-node`'s npm cache. The only thing large enough to be worth caching is `e2e`, and caching a required check's test results on an input hash is a protection decision, not a tooling one. The planner already makes that bet explicitly and reviewably.
Contributor guide
Assessment
This issue has not been assessed yet.