[P2] Restore benchmark gate signal: publish a mocked series and re-derive ceilings from CI data

**Parent:** [#7207 Statistical Sanitization](https://github.com/MetaMask/MetaMask-planning/issues/7207)
**Files:** `test/e2e/benchmarks/utils/thresholds.ts`, `.github/scripts/benchmark-stats-commit.sh`, `development/metamaskbot-build-announce/historical-comparison.ts`
**Size:** L | **Hours:** ~16-24

---

## Problem

The benchmark quality gate is not a trustworthy signal, for two independent reasons. Repository state below is read at `main` `0350a01878e0d5e0eb1553b18ca7d653b4157cbb`.

**1. Mocked runs have no baseline.** Only `main` and `release/*` publish to `extension_benchmark_stats`, per the `if` condition on the `store-benchmark-stats` job in [`run-benchmarks.yml`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/.github/workflows/run-benchmarks.yml#L320-L332). With populations kept apart, mocked PR runs resolve no baseline, so PR comments show no historical deltas. Reporting nothing is correct — comparing a mocked measurement against a live baseline describes upstream latency rather than the commit — but it is a real loss of signal.

**2. The ceilings were never derived from the population they gate.** [#42291 recalibrate `total` thresholds to Sentry baselines](https://github.com/MetaMask/metamask-extension/pull/42291), merged 2026-05-01, set the four `total` values in `THRESHOLD_REGISTRY` — `onboardingImportWallet`, `onboardingNewWallet`, `importSrpHome`, `swap` — from Sentry 30-day production p75/p95. That is a third population: production is heterogeneous devices and networks with a heavy right tail; CI is one machine class with n≈10–30 reps. A quantile of one is not a quantile of the other. Two defects follow:

- **Referent mismatch.** Sentry p75 is over user sessions; the CI p75 is over repetitions within a single run; [`buildMetricBaselines`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/development/metamaskbot-build-announce/historical-comparison.ts#L135-L163), called from `aggregateHistoricalData`, then averages within-run p75s across commits (`calculateMean(values.p75)`). A mean of p75s is not the p75 of the pooled distribution — quantiles do not commute with averaging. The same function averages per-run `stdDev` over a different set of runs than it averages `mean` and `p75` over, which is tracked separately in [MetaMask-planning#7204](https://github.com/MetaMask/MetaMask-planning/issues/7204).
- **Fixed constants over a drifting series end in permanent red.** `onboardingImportWallet.total` breached on 100% of recorded `main` runs from 2026-06-09 onward — seven weeks before [#45046 benchmark `quality-gate` red on every `main` commit](https://github.com/MetaMask/metamask-extension/issues/45046) was opened on 2026-07-30 to record the standing failure.

**Both halves are still fully open.** The attempt to keep the live population out of the gate ([#45204](https://github.com/MetaMask/metamask-extension/pull/45204)) was closed unmerged on 2026-08-13 with the comment `Closing. Pivoting to using mocks for 100% of our benchmark runs.` The attempt to cap the stats series and unmask its publish failures ([#45445](https://github.com/MetaMask/metamask-extension/pull/45445)) was closed unmerged on 2026-09-01, so [#45451 `main` benchmark baseline publishing silently stopped at GitHub's 100 MB file size limit](https://github.com/MetaMask/metamask-extension/issues/45451) is unaddressed: `benchmark-stats-commit.sh` still has no series cap, and both `continue-on-error: true` masks remain, on the [`store-benchmark-stats` job](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/.github/workflows/run-benchmarks.yml#L322) and on the [commit step](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/.github/workflows/run-benchmarks.yml#L365).

Calibration of `onboardingImportWallet.total` was never blocked on #45266 and still is not — its 862ms between-run sd is a clean, mixture-free measurement.

---

## Measurement

n=30 PR runs, `chrome-webpack`, mocked population, 2026-08-05 16:21–20:21 UTC, from the `benchmark-chrome-webpack-userJourneyOnboardingImport` / `-OnboardingNew` artifacts with the runner image read off each producing job.

### Where the current ceilings sit

`onboardingImportWallet.total`, empirical quantiles: Q0.50 = 10495, Q0.75 = 10778, Q0.90 = 11465, Q0.95 = 11665, Q0.99 = 11718ms.

| ceiling | value | position | fires on |
|---|---:|---|---|
| warn p75 | 9100ms | Q0.07 | 28/30 = **93%** — carries no information |
| fail p75 | 11050ms | **Q0.87** | 4/30 = **13%** of clean PRs |

Both derive from [`thresholds.ts` L85-L89](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/thresholds.ts#L85-L89) — p75 warn 7000 / fail 8500 times `CI_MULTIPLIER.TIER_1` = 1.3 — and both are unchanged at the sha above.

Median 10495ms, MAD 171ms → σ_robust 254ms, against an **actual sd of 862ms**. MAD understates the scale 3.4×, so a median/MAD-derived ceiling lands ~11257 and fires on ~13% of clean runs. The distribution is unimodal with a dense core and heavy tails on both sides.

### The two flows have different problems

| | `onboardingNewWallet.total` | `onboardingImportWallet.total` |
|---|---|---|
| within-run sd | 3279ms | 120ms |
| bimodal | yes | **no** — absorbed by the container |
| between-run sd | dominated by the mixture | **862ms** (8.2% CV), mixture-free |
| fixed by #45266 (the bimodal slow path) | the whole problem | **nothing** |

Both flows end on the same Solana token wait, but the import flow has a step *after* it and the new-wallet flow does not. In the import flow the ~5.6s moves between two adjacent labels and `total` absorbs it either way. In the new-wallet flow `doneButtonToAssetList` is terminal, so the bimodality propagates into `total`.

Consequence: the import flow's 862ms is already a clean measurement of residual noise, and calibration is the only lever there. That number will not move when #45266 lands.

### `onboardingNewWallet.total` is a binomial draw, not a measurement

| cluster | n | median | range | condition |
|---|---:|---:|---|---|
| PASS | 3 | 2265ms | 2257–2300 | `nSlow ≤ 1` |
| FAIL | 11 | 9233ms | 6858–10478 | `nSlow ≥ 2` |

Gap 4558ms, **the 5460ms fail ceiling sits inside it**. Separation by `nSlow` is perfect across the 13 runs where iteration counts were recoverable. Observed failure rate 11/14 = 79%.

Per iteration, `doneButtonToAssetList` is ~1.7–2.5s or ~8.4–10.2s with nothing between; p(slow) = 24/65 = **37%**, indistinguishable from the import flow's 30/80 = 37.5%. Run-level p75 over 5 samples is the 4th order statistic, so it flips at 2 slow draws.

This metric has since been taken off the gate allowlist by [#45444 demote the three bimodal onboarding metrics](https://github.com/MetaMask/metamask-extension/pull/45444), merged 2026-08-28, with the restore condition recorded in [`gated-metrics.ts`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/gated-metrics.ts#L52-L63). It is no longer a source of blocked PRs; it is still uncalibratable, which is what this ticket has to answer.

Outlier trimming makes the split discontinuous rather than merely noisy:

| run | iterations (s) | trimmed | p75 total |
|---|---|---:|---:|
| [31035768357](https://github.com/MetaMask/metamask-extension/actions/runs/31035768357) | 1.87, 1.80, 1.79, 1.91, 1.85 | 0 | 2265 |
| [31038082343](https://github.com/MetaMask/metamask-extension/actions/runs/31038082343) | 2.01, **9.30**, 1.90, 1.73, 1.69 | 1 | 2300 |
| [31043191301](https://github.com/MetaMask/metamask-extension/actions/runs/31043191301) | 2.43, **8.94**, **8.84**, 1.65, 1.62 | **0** | 9233 |
| [31043480002](https://github.com/MetaMask/metamask-extension/actions/runs/31043480002) | **10.17**, 1.67, 1.67, **10.08**, 1.72 | **0** | 10478 |

One slow draw is trimmed away; two widen the outlier boundary until nothing is trimmed.

### `doneButtonToHomeScreen` measures boundary placement, not duration

It and `openAccountMenuToAccountListLoaded` are two labels for one span. Whenever one is slow the other collapses to ~1.05s, and the sum is flat (sd ≈ 0.2s):

| run | `doneButtonToHomeScreen` | `openAccountMenuToAccountListLoaded` | sum |
|---|---|---|---|
| [31026037892](https://github.com/MetaMask/metamask-extension/actions/runs/31026037892) | 3.38, 3.48, 3.34, **9.72**, 3.19 | 7.87, 7.51, 7.88, **1.05**, 8.15 | 11.25, 10.99, 11.22, 10.77, 11.34 |
| [31042229025](https://github.com/MetaMask/metamask-extension/actions/runs/31042229025) | 3.35, 3.42, **8.94**, 3.24, **8.77** | 6.82, 6.87, **1.05**, 6.85, **1.04** | 10.17, 10.29, 9.99, 10.09, 9.81 |

Across all 30 runs, median within-run sd: `doneButtonToHomeScreen` 2662ms, `openAccountMenuToAccountListLoaded` 2674ms, `total` 120ms (< 400ms in 30/30). Near-identical spread in two components against a stable container is the signature of two variables trading a fixed quantity.

The steps do partition `total` per iteration: `total` is the per-run sum of the flow's non-unit step timers, at [`test/e2e/benchmarks/utils/runner.ts` L236-L253](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/runner.ts#L236-L253). What does not partition is the *percentile*: `p75(doneButtonToHomeScreen) + p75(openAccountMenuToAccountListLoaded)` exceeds `p75(total)` in 16 of 30 runs, which is a percentile-addition artifact rather than an overlap. The two steps are anti-correlated within a run, so each one's p75 falls on a different iteration, and percentiles are not additive.

Per iteration the arithmetic holds. In run 31042229025, `done + acct` is 10.17, 10.29, 9.99, 10.09 and 9.81s against a `total` spanning 10.27-10.75s, the remainder being the flow's five other steps.

### Ruled out, and constraints on any future method

- **Runner image is not a splitting variable** — `20260720.247.2` on all 29 runs that resolved a label. It does not vary.
- **More repetitions cannot help.** Within-run sd is 120ms against between-run 862ms. σ for the gate is the between-run component; reps only shrink the within-run one. Only cross-run or cross-commit aggregation moves it.
- **Minimum detectable effect is 20–32% of the median** for `onboardingImportWallet.total` — 2144ms at α=0.05, 2731ms at α=0.01, 3390ms at α=0.001, all at power 0.8. Anything smaller is invisible at one run per commit whatever ceiling is chosen. An MDE against a bimodal null is not meaningful, so this figure does not transfer to `onboardingNewWallet.total`.

---

## Solution

### Publish a mocked series

Run the gated matrix mocked on `main` and publish it as the baseline PRs compare against.

Two prerequisites, neither of which is on `main`:

- **Publishing has to work at all.** #45451 is unfixed and its masks are still in place, so a failed publish is silent. Nothing downstream is testable until that is resolved.
- **Population selection has to exist.** There is no `mockMode` field anywhere in the repository at the sha above — a code search returns no hits, against a control search for `aggregateHistoricalData` that returns its two files — so neither the writer nor `aggregateHistoricalData` can distinguish a mocked entry from a live one. That selection has to be built here; it is not an already-landed dependency.

Whether a live matrix continues as a separate non-gated series is an open decision, not a detail: #45204 was closed in favor of running mocks for 100% of benchmark runs, which would remove the live series rather than demote it.

### Derive ceilings from the CI null distribution

The level must come from the population being gated. Sentry keeps two legitimate roles, neither of them the level:

- **Metric selection** — production volume and p75 say which flows have user impact and deserve a gate at all.
- **Defining Δ** — "a regression worth blocking moves production p75 by X ms". Converting X into CI units needs a *measured* transfer function: regress the CI statistic against the production statistic across past releases. If that relationship is unstable — expected for anything network-bound — the instability is itself the finding, and the metric should not be ceiling-gated.

1. Collect the per-run statistic over K recent clean `main` commits. Use a robust location/scale, but **validate the scale estimator against the empirical spread** before trusting it.
2. Set ceiling = **Q₁₋α of that null distribution** against an explicit false-positive budget. Use the empirical quantile with a bootstrap CI, or a peaks-over-threshold GPD fit for the tail — not a normal-theory z-score, since these distributions are right-skewed and the normal approximation understates the tail.
3. Choose α from run volume. At ~25 `main` runs/day, α=0.01 is a false red every four days. Either lower α or add a **confirmation rule** — fail only on two consecutive breaching runs, which squares the FP rate at the cost of one commit of latency.
4. **Check the gate can do its job** before calibrating it: compute the MDE per metric. For some the honest answer is "not gateable at this sample size".
5. For drifting metrics, prefer **EWMA/CUSUM changepoint detection** over a fixed ceiling. A step-size alarm is scale-free and self-relocating, and answers "did a step land at commit X" — the question a merge gate actually needs.
6. Whatever ceiling is derived, it has to survive the run-dependent adjustments already applied between the configured value and the blocking decision: CV-adaptive widening and the within-noise downgrade. Those are tracked in [#45544](https://github.com/MetaMask/metamask-extension/issues/45544) and [#45935](https://github.com/MetaMask/metamask-extension/issues/45935); a ceiling calibrated to a false-positive budget does not hold if a later multiplier moves it per run.

### Fix the metrics that are not measuring what they claim

`doneButtonToHomeScreen` should be merged with its neighbour, given a deterministic boundary, or dropped from the allowlist — not re-thresholded. Structural half tracked in [#45452 benchmark step timers are not a partition of the flow they measure](https://github.com/MetaMask/metamask-extension/issues/45452); mechanism half in [#45266](https://github.com/MetaMask/metamask-extension/issues/45266). It is already off the gate allowlist as of #45444, so this is about the measurement, not about unblocking PRs.

---

## Acceptance Criteria

- [ ] `main` baseline publishing is verified working end to end, with the failure no longer masked
- [ ] Population selection exists: mocked and live entries are distinguishable by the writer and by `aggregateHistoricalData`
- [ ] A mocked benchmark series is published and PR runs resolve a mocked baseline
- [ ] PR-comment deltas are restored and compare mocked-to-mocked
- [ ] Every gated metric's ceiling is derived from CI data with its α, window, and scale estimator recorded — and the estimator validated against the empirical spread
- [ ] Minimum detectable effect computed per gated metric; metrics below the detection floor are removed from `GATED_METRIC_VALUES` rather than gated with thresholds that cannot work
- [ ] The mean-of-p75s aggregation in `historical-comparison.ts` is replaced or explicitly justified — shared with MetaMask-planning#7204, and closing it once closes it for both
- [ ] Drift-prone metrics use changepoint detection instead of a fixed ceiling
- [ ] `doneButtonToHomeScreen` is merged with its neighbour, given a deterministic boundary, or removed from `THRESHOLD_REGISTRY`
- [ ] `onboardingNewWallet.total` calibrated **only after** #45266 lands and its residual noise floor is measured over ≥20 clean runs. The only mixture-free evidence in the n=30 sample above is one all-fast run (within-run sd 50ms) and three PASS-cluster runs (between-run sd 23ms) — not a distribution, and silent on the tails a ceiling must clear
- [ ] `quality-gate` verified green on `main` for a full day with all metrics re-gated
- [ ] Phase 4 — `continue-on-error: false` plus adding `quality-gate` to branch protection, as described in [`run-benchmarks.yml` L225-L227](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/.github/workflows/run-benchmarks.yml#L225-L227) — can be reconsidered

---

## Dependencies

**Blocked on:** [#45451 `main` benchmark baseline publishing silently stopped at GitHub's 100 MB file size limit](https://github.com/MetaMask/metamask-extension/issues/45451) — open, and its candidate fix #45445 was closed unmerged 2026-09-01
**Blocked on (for `onboardingNewWallet.total` only):** [#45266 onboarding benchmarks are bimodal](https://github.com/MetaMask/metamask-extension/issues/45266) — open; the mock port [#45443](https://github.com/MetaMask/metamask-extension/pull/45443) is still open
**Related:** [#45046 `quality-gate` red on every `main` commit](https://github.com/MetaMask/metamask-extension/issues/45046) — open. [#45452 benchmark step timers are not a partition of the flow they measure](https://github.com/MetaMask/metamask-extension/issues/45452) — open. [#45448 demote three bimodal onboarding metrics out of the enforced set](https://github.com/MetaMask/metamask-extension/issues/45448) — open, though its PR #45444 merged 2026-08-28. [#42291 recalibrate `total` thresholds to Sentry baselines](https://github.com/MetaMask/metamask-extension/pull/42291) — merged 2026-05-01, the source of the ceilings being replaced. [#39587 add mocked HTTP requests and restructure the performance benchmark pipeline](https://github.com/MetaMask/metamask-extension/pull/39587) — merged 2026-02-16. [#43958 complete global non-EVM (Solana) snap-discovery mocks](https://github.com/MetaMask/metamask-extension/issues/43958) — closed 2026-08-04 by [#43961](https://github.com/MetaMask/metamask-extension/pull/43961)

Start with test/e2e/benchmarks/utils/thresholds.ts and gated-metrics.ts to understand the current ceilings and gated metrics, then trace benchmark aggregation in development/metamaskbot-build-announce/historical-comparison.ts. Review .github/scripts/benchmark-stats-commit.sh and the store-benchmark-stats workflow steps to verify how data is published. Done means a distinguishable mocked baseline can be published and benchmark ceilings are derived from the CI population without silent publish failures.

Written by the indexing model from the issue text.