MetaMask / MetaMask/metamask-extension
[P1] Investigate `onboardingImportWallet` benchmark baseline regression (5.5s → 25s)
- Dominant language
- TypeScript
- Stars
- 13.2k
- Forks
- 5.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 451
Description
**Parent Epic:** None — standalone investigation
**File:** `test/e2e/benchmarks/utils/thresholds.ts`
**Size:** M | **Hours:** 6-10
---
## Problem
The gated `onboardingImportWallet` benchmark fails the `run-benchmarks / quality-gate` job on effectively 100% of runs since at least 2026-06-09 — on `main` pushes (e.g. run 27003007890) as well as release PRs (RC #43256, run 27433212146). The gate's constant fail limits were calibrated 2026-05-01 (#42291) against the March–April regime, but the benchmark now runs 4–5× slower:
| Metric | Constant fail limit (p75 × 1.3 CI mult.) | Mar–Apr baseline | Current (main, Jun 9–12) |
| --- | --- | --- | --- |
| `doneButtonToHomeScreen` | 18.2s | ~5.5s median | 22–26s median, 15–41s scatter |
| `total` (p75) | 11.1s | ~8s median | ~30s median, 16–55s scatter |
The shift happened somewhere between **Apr 2 and Jun 9** and was masked by an outage in `extension_benchmark_stats` publishing — `stats/main/performance_data.json` has **zero entries Apr 3 → Jun 9**. Additionally, **firefox has no baseline rows at all** in the stats repo, so firefox results are judged against the stale constants alone and can only ever fail.
**What is the constant fail limit?**
`THRESHOLD_REGISTRY` in `thresholds.ts` defines absolute warn/fail values per metric; `compare-benchmarks.ts` exits 1 when a metric in `GATED_METRICS` exceeds `fail × ciMultiplier`, independent of the relative-to-baseline comparison.
---
## Solution
Bisect the regime shift, repair the data pipeline, then recalibrate — in that order, so new constants are derived from trustworthy data.
**Prime suspect for the shift:** #42304 ("enable Assets controller on test build", merged Jun 2). The `doneButtonToHomeScreen` window is exactly the post-onboarding home-screen load, and the **cross-benchmark** `longTaskTotalDuration`/`tbt` inflation (+500–900% across onboardingImport, onboardingNew, importSrpHome, swap) points to a global test-build change rather than a single-flow regression. Real-API user-journey mode compounds run-to-run variance (CV ~25–30%).
**Codebase Context:**
- `test/e2e/benchmarks/utils/thresholds.ts` — `THRESHOLD_REGISTRY`, `CI_MULTIPLIER`
- `test/e2e/benchmarks/utils/gated-metrics.ts` — `GATED_METRICS` allowlist + documented graduation/demotion procedure
- `development/metamaskbot-build-announce/compare-benchmarks.ts` — gate evaluation; "no baseline" path
- `.github/workflows/run-benchmarks.yml` — `store-benchmark-stats` job (main/release only)
---
## Acceptance Criteria
- [ ] Root-cause commit/range for the ~5.5s → ~25s shift identified (confirm or rule out #42304)
- [ ] Determination: test-build artifact vs. user-facing regression — if user-facing, file a product regression ticket
- [ ] `store-benchmark-stats` outage (Apr 3 – Jun 9) root-caused and fixed
- [ ] Firefox baselines present in `extension_benchmark_stats`; `compare-benchmarks.ts` degrades "no baseline" to warn instead of constant-only fail
- [ ] `onboardingImportWallet` thresholds recalibrated per the 30-day graduation procedure (Sentry CV/FP data)
- [ ] Any interim-demoted metrics re-promoted to `GATED_METRICS` once CV < 30% and FP rate < 5%
- [ ] `quality-gate` green on consecutive main pushes for 7 days
---
## Labels
`team-extension-platform`, `area-performance`
---
## Dependencies
**Depends on:** interim demotion PR for `onboardingImportWallet.doneButtonToHomeScreen` / `.total` (to be opened)
**Related:** #42291 (threshold calibration), #42166 (gate enforcement), #42304 (suspect), #43256 (RC where this surfaced)
Contributor guide
Research direction
Start with test/e2e/benchmarks/utils/thresholds.ts and gated-metrics.ts, then read compare-benchmarks.ts and .github/workflows/run-benchmarks.yml to trace threshold evaluation and stats publication. Use the referenced benchmark runs and commits, including #42304, to bisect the shift and investigate the missing Firefox and Apr 3–Jun 9 baselines. Done means the root cause and pipeline issue are addressed, thresholds are recalibrated, and quality-gate remains green for seven days.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, typescript
- Domain
- ci-cd, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100