pmndrs / pmndrs/examples

Deal the e2e shards by measured cost, not by alphabet

Open
#175 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
122
Forks
29
Avg merge
3d 12h
Merged PRs (30d)
2

Description

Follow-up to #173, which sharded the e2e run across eight jobs. The shards are dealt round-robin over the alphabetical list, which is blind to what an example costs — and the spread says so.

What it costs today

Measured on a cold run, one example at a time per shard (run 31507354115):

shard 1 2 3 4 5 6 7 8
duration 9 min 9 min 7 min 13 min 8 min 5 min 16 min 16 min
  • ~83 min of work in total
  • a perfectly balanced run would be 10.4 min
  • the wall clock is the slowest shard: 16 min, a 3.2× spread between fastest and slowest

Round-robin was chosen because alphabetical neighbours are often the same kind of scene (gltf-animations, gltf-animations-re-used, …), so contiguous blocks would clump the heavy ones. It spreads that, and nothing else: an example costs anywhere from a few seconds to three minutes, and dealing them out one by one is still a coin toss.

Three levers, increasing in value

  • A dynamic matrix, which is nearly free. A setup job computes the matrix and exposes it as an output; the test job takes strategy.matrix: ${{ fromJson(needs.setup.outputs.matrix) }}. This also removes a duplication that has to be maintained by hand today: matrix cannot read env, so SHARDS: 8 and shard: [1..8] are two places saying the same number, and .github/workflows/flaky.yml repeats both.

  • More, smaller shards — the poor version. The smaller a shard, the less imbalance costs: sixteen shards of ~10 examples would put the wall clock near 8 min. But each shard re-pays ~2 min of fixed setup (checkout, pnpm install, turbo cache restore), so the return collapses quickly. Parallelism bought with waste.

  • Deal by measured cost — the real one. Sort by duration descending, give each example to the least-loaded shard (longest-processing-time-first). On the numbers above that brings the wall clock to ~11 min at eight shards, without a single extra CI minute.

Where the durations come from

We already produce most of it: the nightly Flaky workflow writes a JSON verdict per shard. Add a per-example duration to it, publish it as an artifact from main, and have the setup job read the most recent one — falling back to today's round-robin when it is missing, so a first run or a lost artifact degrades rather than breaks.

That makes the split self-calibrating: an example that becomes slow redistributes itself, and nobody has to remember a number.

Not worth it

Work stealing — shards pulling the next example off a shared queue — is the textbook answer and needs a coordinator GitHub Actions does not have. Not at that price, for five minutes.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .github/workflows/flaky.yml and the existing nightly Flaky workflow that writes per-shard JSON verdicts. Trace how the current eight-shard matrix is defined and how artifacts are published, then review GitHub Actions job outputs and matrix syntax. Done means the setup job can use recent per-example durations, fall back when data is missing, and produce a more balanced e2e split.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.