Deal the e2e shards by measured cost, not by alphabet
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 122
- Forks
- 29
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 2
Description
Follow-up to #173, which sharded the e2e run across eight jobs. The shards are dealt round-robin over the alphabetical list, which is blind to what an example costs — and the spread says so.
What it costs today
Measured on a cold run, one example at a time per shard (run 31507354115):
| shard | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| duration | 9 min | 9 min | 7 min | 13 min | 8 min | 5 min | 16 min | 16 min |
- ~83 min of work in total
- a perfectly balanced run would be 10.4 min
- the wall clock is the slowest shard: 16 min, a 3.2× spread between fastest and slowest
Round-robin was chosen because alphabetical neighbours are often the same kind of scene (gltf-animations, gltf-animations-re-used, …), so contiguous blocks would clump the heavy ones. It spreads that, and nothing else: an example costs anywhere from a few seconds to three minutes, and dealing them out one by one is still a coin toss.
Three levers, increasing in value
-
A dynamic matrix, which is nearly free. A
setupjob computes the matrix and exposes it as an output; the test job takesstrategy.matrix: ${{ fromJson(needs.setup.outputs.matrix) }}. This also removes a duplication that has to be maintained by hand today:matrixcannot readenv, soSHARDS: 8andshard: [1..8]are two places saying the same number, and.github/workflows/flaky.ymlrepeats both. -
More, smaller shards — the poor version. The smaller a shard, the less imbalance costs: sixteen shards of ~10 examples would put the wall clock near 8 min. But each shard re-pays ~2 min of fixed setup (checkout,
pnpm install, turbo cache restore), so the return collapses quickly. Parallelism bought with waste. -
Deal by measured cost — the real one. Sort by duration descending, give each example to the least-loaded shard (longest-processing-time-first). On the numbers above that brings the wall clock to ~11 min at eight shards, without a single extra CI minute.
Where the durations come from
We already produce most of it: the nightly Flaky workflow writes a JSON verdict per shard. Add a per-example duration to it, publish it as an artifact from main, and have the setup job read the most recent one — falling back to today's round-robin when it is missing, so a first run or a lost artifact degrades rather than breaks.
That makes the split self-calibrating: an example that becomes slow redistributes itself, and nobody has to remember a number.
Not worth it
Work stealing — shards pulling the next example off a shared queue — is the textbook answer and needs a coordinator GitHub Actions does not have. Not at that price, for five minutes.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .github/workflows/flaky.yml and the existing nightly Flaky workflow that writes per-shard JSON verdicts. Trace how the current eight-shard matrix is defined and how artifacts are published, then review GitHub Actions job outputs and matrix syntax. Done means the setup job can use recent per-example durations, fall back when data is missing, and produce a more balanced e2e split.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- ci-cd, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100