ci: 15-minute CLI shard ceiling cancels otherwise healthy PR suites
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Summary
The PR CLI coverage matrix has outgrown or intermittently exceeds its 15-minute per-job ceiling. Multiple unrelated PRs on the same base have several shards terminated simultaneously at the limit, producing no Vitest blob and forcing the aggregate `cli-tests` and `checks` jobs red without a test assertion failure.
## Evidence
### PR #9932
- Base: `7601ab27a82dfdb67c933c9e24612e52f8c367e4`
- Run: 32606449660
- Canceled at approximately 15 minutes: shards 1, 2, 5, 7, and 12
- Seven other shards passed.
- Each canceled log ends with `The operation was canceled.` during Vitest, followed by `No files were found` for the report artifact.
### Unrelated PR #9950
- Same base: `7601ab27a82dfdb67c933c9e24612e52f8c367e4`
- Run: 32606363353
- Started about two minutes earlier.
- Canceled at the same approximately 15-minute ceiling: shards 2, 4, 5, 6, 7, and 8
- Six other shards passed.
The different canceled-shard sets on unrelated changes indicate a CI duration/variance problem rather than a deterministic test assertion regression.
## Current configuration
Both `.github/workflows/pr.yaml` and `.github/workflows/main.yaml` set `timeout-minutes: 15` for `cli-test-shards`. The composite action installs dependencies, builds the plugin and CLI, then runs coverage before that shared job deadline.
## Expected behavior
The authoritative broad gate should finish all 12 shard reports with sufficient headroom. Options include increasing the job timeout, moving setup/build outside the measured shard budget, refreshing timing hints/shard allocation, or splitting the heaviest files.
## Validation request
Measure p95 durations for each shard on the current main test roster and set a bounded timeout with operational headroom. Preserve the aggregate fail-closed behavior for genuine test failures.
Contributor guide
Research direction
Start with .github/workflows/pr.yaml and .github/workflows/main.yaml, then inspect the cli-test-shards composite action and recent shard timings. Measure p95 duration for each of the 12 shards on the current main test roster. Done means the authoritative gate completes all shard reports with bounded timeout headroom while still failing for genuine test failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, typescript
- Domain
- ci-cd, cli, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100