MetaMask / MetaMask/metamask-extension

[P1] Investigate `onboardingImportWallet` benchmark baseline regression (5.5s → 25s)

Open
#43,518 0 comments 0 reactions 0 assignees View on GitHub
area-performance INVALID-ISSUE-TEMPLATE Sev2-normal team-assets team-extension-platform
Dominant language
TypeScript
Stars
13.2k
Forks
5.6k
Avg merge
2d 5h
Merged PRs (30d)
451

Description

**Parent Epic:** None — standalone investigation
**File:** `test/e2e/benchmarks/utils/thresholds.ts`
**Size:** M | **Hours:** 6-10

---

## Problem

The gated `onboardingImportWallet` benchmark fails the `run-benchmarks / quality-gate` job on effectively 100% of runs since at least 2026-06-09 — on `main` pushes (e.g. run 27003007890) as well as release PRs (RC #43256, run 27433212146). The gate's constant fail limits were calibrated 2026-05-01 (#42291) against the March–April regime, but the benchmark now runs 4–5× slower:

| Metric | Constant fail limit (p75 × 1.3 CI mult.) | Mar–Apr baseline | Current (main, Jun 9–12) |
| --- | --- | --- | --- |
| `doneButtonToHomeScreen` | 18.2s | ~5.5s median | 22–26s median, 15–41s scatter |
| `total` (p75) | 11.1s | ~8s median | ~30s median, 16–55s scatter |

The shift happened somewhere between **Apr 2 and Jun 9** and was masked by an outage in `extension_benchmark_stats` publishing — `stats/main/performance_data.json` has **zero entries Apr 3 → Jun 9**. Additionally, **firefox has no baseline rows at all** in the stats repo, so firefox results are judged against the stale constants alone and can only ever fail.

**What is the constant fail limit?**
`THRESHOLD_REGISTRY` in `thresholds.ts` defines absolute warn/fail values per metric; `compare-benchmarks.ts` exits 1 when a metric in `GATED_METRICS` exceeds `fail × ciMultiplier`, independent of the relative-to-baseline comparison.

---

## Solution

Bisect the regime shift, repair the data pipeline, then recalibrate — in that order, so new constants are derived from trustworthy data.

**Prime suspect for the shift:** #42304 ("enable Assets controller on test build", merged Jun 2). The `doneButtonToHomeScreen` window is exactly the post-onboarding home-screen load, and the **cross-benchmark** `longTaskTotalDuration`/`tbt` inflation (+500–900% across onboardingImport, onboardingNew, importSrpHome, swap) points to a global test-build change rather than a single-flow regression. Real-API user-journey mode compounds run-to-run variance (CV ~25–30%).

**Codebase Context:**
- `test/e2e/benchmarks/utils/thresholds.ts` — `THRESHOLD_REGISTRY`, `CI_MULTIPLIER`
- `test/e2e/benchmarks/utils/gated-metrics.ts` — `GATED_METRICS` allowlist + documented graduation/demotion procedure
- `development/metamaskbot-build-announce/compare-benchmarks.ts` — gate evaluation; "no baseline" path
- `.github/workflows/run-benchmarks.yml` — `store-benchmark-stats` job (main/release only)

---

## Acceptance Criteria

- [ ] Root-cause commit/range for the ~5.5s → ~25s shift identified (confirm or rule out #42304)
- [ ] Determination: test-build artifact vs. user-facing regression — if user-facing, file a product regression ticket
- [ ] `store-benchmark-stats` outage (Apr 3 – Jun 9) root-caused and fixed
- [ ] Firefox baselines present in `extension_benchmark_stats`; `compare-benchmarks.ts` degrades "no baseline" to warn instead of constant-only fail
- [ ] `onboardingImportWallet` thresholds recalibrated per the 30-day graduation procedure (Sentry CV/FP data)
- [ ] Any interim-demoted metrics re-promoted to `GATED_METRICS` once CV < 30% and FP rate < 5%
- [ ] `quality-gate` green on consecutive main pushes for 7 days

---

## Labels

`team-extension-platform`, `area-performance`

---

## Dependencies

**Depends on:** interim demotion PR for `onboardingImportWallet.doneButtonToHomeScreen` / `.total` (to be opened)
**Related:** #42291 (threshold calibration), #42166 (gate enforcement), #42304 (suspect), #43256 (RC where this surfaced)

Contributor guide

Open the contributing guide

Research direction

Start with test/e2e/benchmarks/utils/thresholds.ts and gated-metrics.ts, then read compare-benchmarks.ts and .github/workflows/run-benchmarks.yml to trace threshold evaluation and stats publication. Use the referenced benchmark runs and commits, including #42304, to bisect the shift and investigate the missing Firefox and Apr 3–Jun 9 baselines. Done means the root cause and pipeline issue are addressed, thresholds are recalibrated, and quality-gate remains green for seven days.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, performance, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.