Skill: benchmark a toolchain change across build, CI, and runtime

**Size:** L | **Domain:** `platform` (or a new `build`)

---

## Problem

A toolchain change moves three independent numbers, and measuring one misleads about the others:

| Surface | `React Compiler` | `thread-loader` | `oxlint` |
|---|---|---|---|
| Build time (dev loop) | **slower** — extra transform | **faster** — parallel pool | faster than `ESLint` |
| CI time | slower | depends on runner cores | faster |
| Runtime performance | **faster** — memoized output | neutral — output unchanged | neutral |

Judge `React Compiler` on build time and you reject it. Judge it on runtime and you hide a dev-loop tax every engineer pays on every rebuild. The `webpack` enablement is the flagship case — it moved all three at once, and there was no consistent way to say by how much.

This skill publishes into two evidence lanes — `D7 Build & rebuild duration A/B` for the dev loop and `G6 CI job-duration delta` for CI — which encode the confounds below as their falsifiers. Both are in review on #84, not yet on the default branch. `C5 benchmark A/B` covers runtime only.

---

## Why this is hard to measure honestly

Each falsifier below silently produces a *favourable* result, which is what makes ad-hoc measurement worse than none:

- **Cache state differs between arms.** Warm versus cold is the largest confound and the easiest to introduce by running the new arm second.
- **Worker-pool warmup.** `thread-loader` pays pool startup on the first build and not on rebuilds, so first-build numbers understate it and steady-state numbers overstate it.
- **Core count.** Parallel loaders scale with available cores, so a laptop result does not transfer to a CI runner, or the reverse.
- **Watch rebuild versus cold build** are different questions; conflating them answers neither.
- **CI build reuse.** `get-requirements.yml` skips jobs when build output matches base, so a measured CI speedup can be a reuse-rate change rather than a faster build.
- **Release adoption curve.** On the runtime side, a window sampled before rollout plateaus mixes versions and reads as improvement.

The `thread-loader` integration also carries a coupling worth testing rather than trusting: loader options must stay JSON-serializable for the pool, and the pool must be ordered before the `SWC` rules.

---

## Solution

A paired A/B protocol per surface, each with its confound named and controlled:

1. **Dev loop** — cold and warm reported separately, cache cleared explicitly between arms, N runs reported with spread rather than a mean alone.
2. **CI** — same workflow, same runner class, arms at a fixed commit differing only by the flag, with reuse rate stated so a skip is not read as a speedup.
3. **Runtime** — p75/p95 over matched windows after adoption plateaus, with the version filter stated.

The verdict is a table across all three. A change that costs build time and buys runtime is a **trade**; the skill prices it rather than approving it.

---

## Acceptance criteria

- [ ] A protocol per surface, each stating its controls
- [ ] A falsifier per arm: cache state, pool warmup, core count, CI build reuse, adoption window
- [ ] Worked example on the `React Compiler` / `thread-loader` enablement, real numbers, both arms
- [ ] Reports spread, not central tendency alone; states the smallest detectable effect when a result is null
- [ ] Renders no ship verdict — the trade-off belongs to whoever owns the dev loop

---

## Notes

May want splitting once the shape is clear: three protocols under one skill is a lot, and the dev-loop and CI halves may separate cleanly from the runtime half.

Related but distinct from a CI-health cluster. This prices a **known** change across surfaces; diagnosing an **unexplained** CI regression is the other direction and wants different instruments.

---

Tracked in Jira as `AIENG-95`.

Start by reading the D7 Build & rebuild duration A/B and G6 CI job-duration delta lanes in review on #84, along with get-requirements.yml; compare them with C5 benchmark A/B to define the three protocol boundaries. Done means each surface has controlled arms, named falsifiers, spread and null-result reporting, and a worked React Compiler/thread-loader example with real numbers and no ship verdict.

Written by the indexing model from the issue text.