MetaMask / MetaMask/metamask-extension

`onboardingImportWallet.total` breaches its ceiling on `main`, traced to the `@sentry/browser` 10.x bump

Open
#45,046 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

INVALID-ISSUE-TEMPLATE Sev2-normal ta-ai-fixable ta-needs-engineer-escalation ta-triaged team-qa
Dominant language
TypeScript
Stars
13.2k
Forks
5.6k
Avg merge
2d 5h
Merged PRs (30d)
451

Description

### Summary

The benchmark `quality-gate` job failed on every `main` commit from `7bfc16cfc0d` (2026-07-17), and therefore on every PR branched from `main`, with the identical `FAIL onboardingImportWallet` + `FAIL onboardingNewWallet` pair. That unconditional red has ended. The failure has not: sampled 2026-09-02, `quality-gate` failed on 3 of the 12 most recent completed `main` push runs, and all three failures are `FAIL onboardingImportWallet [chrome-webpack]` on `total`.

The `onboardingNewWallet` half stopped appearing because [#45444 (demote the three bimodal onboarding metrics out of the gate)](https://github.com/MetaMask/metamask-extension/pull/45444), merged 2026-08-28, took it out of `GATED_METRICS`. `onboardingImportWallet.total` was deliberately kept gated there, on an 8.2% CV, and it is what still reds `main`.

The original diagnosis was that the cause is not a product regression: three gated onboarding metrics terminate on a `waitForTokenToBeDisplayed('Solana', …)` call, so they were measuring the unmocked Solana snap-discovery retry storm tracked in #43958 — roughly 6.5 seconds of waiting — rather than extension performance. That diagnosis is at best incomplete. [#43958 (complete the global non-EVM Solana snap-discovery mocks)](https://github.com/MetaMask/metamask-extension/issues/43958) closed `completed` on 2026-08-04 and the gate went on failing. [#45266 (onboarding benchmarks are bimodal: a ~7s slow path on a large minority of iterations)](https://github.com/MetaMask/metamask-extension/issues/45266), open, carries the competing account, and it is the one #45444 acted on.

### Evidence

- **The gate was red continuously on `main` at filing (2026-07-30).** Eight consecutive `main` commits, all `quality-gate=failure` with the identical `FAIL onboardingImportWallet` + `FAIL onboardingNewWallet` pair: `9afac38fe`, `bfdb62afa`, `0fdbade0f`, `33bef9f69`, `6d53060f6`, `ed7b8fbd5`, `7e9653a51`, `5a94f11f6`.
- **Still red a month later.** Twelve completed `main` push runs spanning 2026-08-19T15:32Z to 2026-08-20T11:49Z, read job-by-job with `gh api "repos/MetaMask/metamask-extension/actions/runs//jobs?per_page=100" --paginate --jq '.jobs[] | select(.name|test("quality-gate")) | .conclusion'`: 12 of 12 `failure` (runs `32270662173`, `32280876016`, `32284973429`, `32288011379`, `32288568812`, `32293677758`, `32297236150`, `32304813359`, `32316226930`, `32318877573`, `32351557269`, `32365675817`).
- **Now intermittent, not standing.** The same read over the 12 most recent completed `main` push runs, 2026-09-01T20:23Z to 2026-09-02T13:22Z, sampled 2026-09-02: **9 `success`, 3 `failure`** — runs [33568042742](https://github.com/MetaMask/metamask-extension/actions/runs/33568042742), [33568606310](https://github.com/MetaMask/metamask-extension/actions/runs/33568606310) and [33627544662](https://github.com/MetaMask/metamask-extension/actions/runs/33627544662). Every one of the three prints `FAIL onboardingImportWallet [chrome-webpack]` and no other benchmark. From run 33627544662, `quality-gate` job `100246174518`:

```
FAIL onboardingImportWallet [chrome-webpack]
🔴 total | p75: 11193ms (absolute ceiling exceeded, limit 11050ms) | p95: 11223ms (absolute ceiling exceeded, limit 9750ms)
RESULT: FAIL — at least one benchmark exceeds constant fail limit
```

- These are two 12-run samples, not a census, and they do not date the transition from standing to intermittent. #45444's merge on 2026-08-28 accounts for the `onboardingNewWallet` half disappearing; nothing here establishes what changed for `onboardingImportWallet`.
- **The breach was not marginal or noisy.** Across [`extension_benchmark_stats` `stats/main/performance_data.json`](https://github.com/MetaMask/extension_benchmark_stats/blob/2b0cdd015e1b6fce43dc4113a8257f05f712b6b1/stats/main/performance_data.json):
- `onboardingNewWallet.total` p75 breached its 5,460 ms ceiling (`4200 × CI_MULTIPLIER.TIER_1`) in **76 of 77** `main` runs over the 7 days to 2026-07-30, median **9,827 ms** — 1.8× the ceiling.
- Within a single run the spread is small (stdDev 165 ms in [run 30546745479](https://github.com/MetaMask/metamask-extension/actions/runs/30546745479)), so noise tolerance and the CV-adaptive widening from [#44609 (harden the absolute benchmark gate against within-noise breaches)](https://github.com/MetaMask/metamask-extension/pull/44609), merged 2026-07-22, could not absorb it.
- That series is no longer current, which is #45451 (`main` baseline publishing stopped at the 100 MiB limit): `stats/main/performance_data.json` has taken no commit since 2026-07-28T09:05:23Z (`7260d170e`), while the same query over `stats/` on the same branch returns commits through 2026-09-01. The link therefore shows a series ending 2026-07-28.
- **The changepoint isolates to one commit.** Two adjacent `main` data points, with exactly one commit between them:
- `a25c6a5b07b43883a6c03a042ae04d3f5df48654` (07-17 11:30) — `doneButtonToAssetList` p75 = **3,514 ms**
- `7bfc16cfc` (07-17 12:02) — `doneButtonToAssetList` p75 = **9,573 ms**
- The GitHub compare API, with base `a25c6a5b07b43883a6c03a042ae04d3f5df48654` and head `7bfc16cfc0df76079344272ccd1cde3c880b12fe`, returns `total_commits: 1`, that commit being #42867 (`@sentry/browser` 8.33.1 → 10.38.0).

### Affected metrics

Median p75 over the 9 days before the changepoint vs. the 13 days after, measured at filing (2026-07-30):

| benchmark · metric | before | after | Δ |
|---|---:|---:|---:|
| `importSrpHome.homeAfterImportWithNewWallet` | 12,416 | 27,724 | +123% |
| `onboardingImportWallet.doneButtonToHomeScreen` | 25,418 | 36,836 | +45% |
| `onboardingImportWallet.openAccountMenuToAccountListLoaded` | 2,308 | 9,274 | +302% |
| `onboardingNewWallet.doneButtonToAssetList` | 3,118 | 9,650 | +210% |
| `importSrpHome.openAccountMenuAfterLogin` | 57 | 864 | +1404% |
| `loadNewAccount.load_new_account` | 275 | 1,076 | +291% |

### Why this is a measurement artifact, not a regression

- **The affected set is exactly the non-EVM waits.** The three largest movers each end a measured step on a Solana token wait, all three still present on `main` at `0350a0187`:
- [`onboarding-new-wallet.ts#L163`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/flows/user-journey/onboarding-new-wallet.ts#L163) — `waitForTokenToBeDisplayed('Solana', 60000)`, closing `doneButtonToAssetList`
- [`onboarding-import-wallet.ts#L184`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/flows/user-journey/onboarding-import-wallet.ts#L184) — `waitForTokenToBeDisplayed('Solana', 120000)`, closing `doneButtonToHomeScreen`
- [`import-srp-home.ts#L109`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/flows/user-journey/import-srp-home.ts#L109) — `waitForTokenToBeDisplayed('Solana', 60000)`, closing `homeAfterImportWithNewWallet`
- **The added time is waiting, not work.** Over the same boundary, `onboardingNewWallet` `tbt` fell 154 → 101 ms and `longTaskTotalDuration` fell 326 → 275 ms. CPU work went slightly *down* while wall-clock went up 6.5 s.
- **The mechanism is already filed.** #43958 documents that 14 of the Solana snap's 15 discovery RPC methods fall through to the empty-200 catch-all, producing a ~516-request retry storm that pushes the `solana-logo` icon past the default wait. #42867 amplified it — its own description records it as *"Surfaced (not caused) by removing the e2e startup band-aids"* — and shipped `NON_EVM_ICON_TIMEOUT` in [`homepage.ts:23`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/page-objects/pages/home/homepage.ts#L23) as an interim widening for the same reason.
- **What closing #43958 did not do.** It closed `completed` on 2026-08-04, and `quality-gate` was still failing on all 12 sampled `main` push runs two weeks later. Its mocks live in `test/e2e/mock-e2e.js`; porting them into the benchmark suite is [#45443](https://github.com/MetaMask/metamask-extension/pull/45443), still open. So the retry storm is not established as the whole cause.
- **Residual uncertainty:** this is CI-build evidence. Whether any part of the delta reproduces against live (non-mocked) Solana RPCs has not been measured, and would need production Sentry data to confirm or rule out. Since 2026-08-24 that question is not askable of CI at all: [#45679 (use mocked requests for benchmarks on `main` and `release/*`)](https://github.com/MetaMask/metamask-extension/pull/45679) made `shouldUseMockedRequests()` return `true` unconditionally ([`mock-config.ts:12-15`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/mock-config.ts#L12-L15)), so every branch runs mocked.

### Impact

- An intermittent red is worse for attribution than a standing one, not better. A standing failure is at least recognizable as not-yours; a gate that reds on one in four `main` pushes for a reason the output does not name is exactly the signal a PR author cannot separate from their own change. That legibility gap is #45934.
- `quality-gate` runs at `continue-on-error: false` — Phase 3 of the rollout described in [`run-benchmarks.yml:223-230`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/.github/workflows/run-benchmarks.yml#L223-L230). Phase 4, adding it to the branch-protection required checks, is the next step in that block and cannot be taken while `main` reds on itself.
- Authors are pointed at `skip-benchmark-gate` for a failure that has nothing to do with their change, which erodes the label's meaning.

### Proposed resolution

- **`onboardingImportWallet.total` is the live failure.** It kept its gate in #45444 on a unimodal 8.2% CV, and it is the metric reding `main` at `p75: 11193ms` against `limit 11050ms` — a 1.3% overshoot, which is a calibration question rather than a regression. It is also named in [`gated-metrics.ts:60-63`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/gated-metrics.ts#L60-L63) as the calibration target of [#45205 (publish a mocked series and re-derive ceilings from CI data)](https://github.com/MetaMask/metamask-extension/issues/45205). Re-deriving that ceiling from a published mocked series is the ordered fix; raising it by hand is not.
- **Three metrics are demoted, not fixed.** #45444 took `onboardingNewWallet.total`, `onboardingNewWallet.doneButtonToAssetList` and `onboardingImportWallet.doneButtonToHomeScreen` out of `GATED_METRICS`, with the restore condition recorded against #45266 in [`gated-metrics.ts:40-64`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/gated-metrics.ts#L40-L64). Closing this ticket on that demotion hides the failure rather than restoring coverage of the onboarding totals.
- **The mocks are not in the benchmark suite yet.** #43958's completed Solana discovery mocks are in `test/e2e/mock-e2e.js`; #45443 ports them into the benchmark suite and is open. Landing it is the test of whether the retry storm was in fact driving the breach.
- **Do not raise the thresholds.** If the measurement is contaminated, a recalibration bakes the retry storm into the baseline and would have to be reverted.
- **Before Phase 4 is reconsidered**, the three demoted metrics should be re-gated and the gate verified green on `main` for a full day with them in place.

### Related

- #43958 — Solana snap-discovery mock gap (closed 2026-08-04; proposed root cause, not confirmed as the whole cause)
- #45266 — onboarding benchmarks are bimodal (open; the competing account, and the one #45444 acted on)
- #45443 — port the completed Solana discovery mocks into the benchmark suite (open)
- #45444 — demote the three bimodal onboarding metrics out of the gate (merged 2026-08-28)
- #45205 — publish a mocked series and re-derive ceilings from CI data (open; owns the `onboardingImportWallet.total` ceiling)
- #45934 — gate output cannot tell a developer whether a red is about their change (open; why an intermittent red is unattributable)
- #42867 — `@sentry/browser` v10 upgrade (changepoint commit; amplifier, not cause)
- #44609 — harden the absolute benchmark gate against within-noise breaches (merged 2026-07-22; does not absorb a breach this large)
- #45451 — `main` baseline publishing stopped at the 100 MiB limit (open; why the linked series stops at 2026-07-28)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read test/e2e/benchmarks/utils/gated-metrics.ts and the quality-gate section of .github/workflows/run-benchmarks.yml, then compare the benchmark mocks in test/e2e/mock-e2e.js with test/e2e/benchmarks/utils/mock-config.ts. Verify the published mocked series and quality-gate runs before deciding whether onboardingImportWallet.total should be recalibrated; done means the gate is green on main without hiding affected metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, performance, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.