MetaMask / MetaMask/metamask-extension
[P1] Onboarding benchmarks are bimodal at ~2s majority vs ~7s large minority
- Dominant language
- TypeScript
- Stars
- 13.2k
- Forks
- 5.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 451
Description
**Parent Epic:** [#6944 Performance Quality Gates](https://github.com/MetaMask/MetaMask-planning/issues/6944) → [#7182 Benchmarks: Reliability](https://github.com/MetaMask/MetaMask-planning/issues/7182) → [#7181 Benchmark Harness Fixes](https://github.com/MetaMask/MetaMask-planning/issues/7181)
**File:** `test/e2e/benchmarks/flows/user-journey/onboarding-new-wallet.ts:163`, `test/e2e/benchmarks/flows/user-journey/onboarding-import-wallet.ts:184` (at `0350a018`)
**Size:** M | **Hours:** ~8-12
---
## Problem
37% of onboarding benchmark iterations take a ~7s slow path that the other 63% do not. This is a per-iteration coin flip, not a property of the branch under test, and it is what makes `onboardingNewWallet.total` unusable as a gate (see [#45205 restore benchmark gate signal](https://github.com/MetaMask/metamask-extension/issues/45205) and the measurements below).
**[#43961 complete Solana discovery RPC mocks](https://github.com/MetaMask/metamask-extension/pull/43961)'s fix was never in this code path.** That PR completed `setupDefaultNonEvmDiscoveryMocks` in `test/e2e/mock-e2e.js` and closed [#43958 complete global non-EVM snap-discovery mocks](https://github.com/MetaMask/metamask-extension/issues/43958) on 2026-08-04 15:12. The sample below was collected 26 hours later and shows the slow path at an unchanged rate — not because the fix failed, but because **the benchmark suite is a separate mock implementation that never loaded that helper.** `test/e2e/benchmarks/` contains no reference to `setupDefaultNonEvmDiscoveryMocks` at `0350a018`.
The unmatched methods reach `solanaCatchAllResponse` ([`mock-responses.ts:510-519`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/mocks/mock-responses.ts#L510-L519)), which returns `{ context, value: null }`: a valid JSON-RPC envelope with the wrong shape for a genesis hash, a version object, a slot number or a health string. That is the malformed-body condition [#43961 complete Solana discovery RPC mocks](https://github.com/MetaMask/metamask-extension/pull/43961) measured, reproduced here.
The gap is still open. At `0350a018` the benchmark suite explicitly matches six of the discovery methods — `getBalance` (`performance-mocks.ts:1093`), `getAccountInfo` (:1106), `getLatestBlockhash` (:1119), `getFeeForMessage` (:1128), `getTokenAccountsByOwner` (:1148), `getSignaturesForAddress` (:1166) — and routes `getGenesisHash`, `getHealth`, `getVersion`, `getSlot`, `getMultipleAccounts`, `getProgramAccounts`, `getTokenAccountBalance`, `getEpochInfo` and `getTransaction` to the catch-all at `performance-mocks.ts:1172-1182`. `test/e2e/mock-e2e.js` covers all fifteen with realistic per-method values (`SOLANA_DISCOVERY_RPC_RESULTS`, lines 173–204).
**[#45443 port the completed Solana discovery mocks into the benchmark suite](https://github.com/MetaMask/metamask-extension/pull/45443) ports the completed result map across, and is open as of 2026-09-02.** It is falsifiable on the next `main` run after it merges: if this is the cause, `doneButtonToAssetList` collapses to its single ~1.7–2.5s mode. If it does not move, the Solana attribution below is wrong and the search reopens.
The duplication itself is tracked as [#45449 some mock suites for e2e and benchmarks are parallel implementations that drift silently](https://github.com/MetaMask/metamask-extension/issues/45449).
### Observation
Raw per-iteration `doneButtonToAssetList` values, read from the `run-benchmarks / chrome-webpack-userJourneyOnboardingNew` job log of each run below. Slow iterations in bold. Every value is recoverable from the linked run.
| run | iterations (s) | slow | `total` p75 (ms) |
|---|---|---:|---:|
| [31043480002](https://github.com/MetaMask/metamask-extension/actions/runs/31043480002) | **10.17**, 1.67, 1.67, **10.08**, 1.72 | 2/5 | 10478 |
| [31043191301](https://github.com/MetaMask/metamask-extension/actions/runs/31043191301) | 2.43, **8.94**, **8.84**, 1.65, 1.62 | 2/5 | 9233 |
| [31042229025](https://github.com/MetaMask/metamask-extension/actions/runs/31042229025) | 2.05, **8.79**, **8.80**, 1.77, 2.53 | 2/5 | 9196 |
| [31039460659](https://github.com/MetaMask/metamask-extension/actions/runs/31039460659) | **9.91**, 1.69, **9.78**, 2.69, 2.07 | 2/5 | 10175 |
| [31038129183](https://github.com/MetaMask/metamask-extension/actions/runs/31038129183) | 2.25, **8.39**, 1.67, **8.73**, **8.38** | 3/5 | 8788 |
| [31038082343](https://github.com/MetaMask/metamask-extension/actions/runs/31038082343) | 2.01, **9.30**, 1.90, 1.73, 1.69 | 1/5 | 2300 |
| [31037888234](https://github.com/MetaMask/metamask-extension/actions/runs/31037888234) | 1.93, **9.00**, 1.68, 2.46, **9.03** | 2/5 | 9415 |
| [31036699656](https://github.com/MetaMask/metamask-extension/actions/runs/31036699656) | 0.92, **6.55**, **6.50**, 5.90, 1.00 | 2/5 | 6858 |
| [31036595608](https://github.com/MetaMask/metamask-extension/actions/runs/31036595608) | **8.64**, **8.52**, 1.73, 1.81, 2.03 | 2/5 | 8918 |
| [31036395392](https://github.com/MetaMask/metamask-extension/actions/runs/31036395392) | **9.05**, 1.85, 1.84, 1.81, 1.74 | 1/5 | 2257 |
| [31036178696](https://github.com/MetaMask/metamask-extension/actions/runs/31036178696) | 2.09, 1.82, **9.52**, **9.24**, **9.20** | 3/5 | 9634 |
| [31036057187](https://github.com/MetaMask/metamask-extension/actions/runs/31036057187) | 2.53, 1.84, **10.22**, **9.79**, 2.36 | 2/5 | 10224 |
| [31035768357](https://github.com/MetaMask/metamask-extension/actions/runs/31035768357) | 1.87, 1.80, 1.79, 1.91, 1.85 | 0/5 | 2265 |
All 13 runs are `pull_request` events on 2026-08-05, i.e. the mocked population, all after [#43961 complete Solana discovery RPC mocks](https://github.com/MetaMask/metamask-extension/pull/43961) merged.
### Measurement
Per-iteration `doneButtonToAssetList` (chrome-webpack, 13 PR runs, 65 iterations):
- Two clean modes: **~1.7–2.5s** and **~8.4–10.2s**, nothing in between.
- **p(slow) = 24/65 = 37%.** The import flow's equivalent step gives 30/80 = 37.5% — the same coin.
- Magnitude of the gap (~5.6–7s) is consistent with the ~6.5s Solana snap-discovery retry storm documented in [#45046 benchmark `quality-gate` red on every `main` commit since 2026-07-17](https://github.com/MetaMask/metamask-extension/issues/45046).
Both affected steps terminate on a Solana token wait:
- `onboarding-new-wallet.ts:163` — `waitForTokenToBeDisplayed('Solana', 60000)`, closing `doneButtonToAssetList`
- `onboarding-import-wallet.ts:184` — `waitForTokenToBeDisplayed('Solana', 120000)`, closing `doneButtonToHomeScreen`
### Why it matters more in one flow than the other
`doneButtonToAssetList` is **terminal** — nothing follows it to absorb the spill — so the bimodality propagates into `total`:
| flow | within-run sd of `total` |
|---|---:|
| `onboardingImportWallet` | 120ms |
| `onboardingNewWallet` | **3279ms** |
`onboardingNewWallet.total` p75 consequently splits into a 2257–2300ms cluster (`nSlow ≤ 1`) and a 6858–10478ms cluster (`nSlow ≥ 2`), with its 5460ms fail ceiling sitting inside the 4558ms gap. Observed failure rate 11/14 = 79%, with no regression involved.
**Confidence note:** the attribution to Solana discovery is strong circumstantial — step boundary, magnitude, and the [#45046 gate red on every `main` commit](https://github.com/MetaMask/metamask-extension/issues/45046) precedent all line up — but the slow path has not been directly instrumented since [#43961 complete Solana discovery RPC mocks](https://github.com/MetaMask/metamask-extension/pull/43961) landed. Confirming *what* the wait is blocked on is the first task below, not an assumption to carry forward.
---
## Solution
1. **Instrument the wait.** Capture, per iteration, what `waitForTokenToBeDisplayed('Solana', …)` blocks on and for how long — request log, retry count, and the token's presence at the moment the wait is entered. Two hypotheses to separate: a surviving mock gap (requests still falling through) versus a timing race that mocks cannot address. Neither is established; the table above shows only the resulting durations.
2. **Fix the mechanism the instrumentation identifies.** If requests are still falling through, close the remaining mock gap. If it is a race between snap discovery completing and the wait being entered, the fix is a deterministic readiness signal rather than more mocks.
3. **Re-measure p(slow).** Target is 0. Anything above a few percent leaves the gated metrics discontinuous.
4. **Re-gate.** Restoring the three metrics demoted below is this ticket's exit condition, not a follow-up.
---
## Acceptance Criteria
- [ ] The slow path's cause is identified from direct instrumentation, not inferred from step boundaries
- [ ] p(slow iteration) measured at < 2% over ≥ 50 iterations on the mocked population
- [ ] `onboardingNewWallet.total` within-run sd back under 400ms, matching the import flow's 120ms order of magnitude
- [ ] `onboardingNewWallet.total` p75 no longer bimodal across ≥ 20 runs
- [ ] **Residual noise floor re-measured after the fix** — unknown for this flow as of 2026-08-06. The only clean evidence is one all-fast run (within-run sd 50ms) and three runs whose p75 landed in the PASS cluster (between-run sd 23ms). That is far too small a sample to assert a floor, and [#45205 restore gate signal](https://github.com/MetaMask/metamask-extension/issues/45205) cannot calibrate a ceiling without it.
- [ ] `onboardingNewWallet.total`, `onboardingNewWallet.doneButtonToAssetList` and `onboardingImportWallet.doneButtonToHomeScreen` are back in `GATED_METRIC_VALUES` in `test/e2e/benchmarks/utils/gated-metrics.ts`, and the three `does not fail on the bimodal metric ...` tests added by [#45444 demote the three bimodal onboarding metrics out of the gate](https://github.com/MetaMask/metamask-extension/pull/45444) (merged 2026-08-28) are inverted to assert the gate fires again. Closing this issue with the gate still off is the failure mode [#45448 demote three bimodal onboarding metrics out of the enforced set](https://github.com/MetaMask/metamask-extension/issues/45448) flagged.
---
## Labels
`team-extension-platform`, `area-testSuite`, `area-CI`
---
## Dependencies
**Blocks:** [#45205 restore benchmark gate signal](https://github.com/MetaMask/metamask-extension/issues/45205) — ceiling calibration for `onboardingNewWallet.total` is not meaningful until the null is unimodal.
**Related:** [#45046 gate red on every `main` commit](https://github.com/MetaMask/metamask-extension/issues/45046), [#43958 complete global non-EVM snap-discovery mocks](https://github.com/MetaMask/metamask-extension/issues/43958) (closed by [#43961 complete Solana discovery RPC mocks](https://github.com/MetaMask/metamask-extension/pull/43961)), [#45204 stop gating the live-network benchmark population against mocked ceilings](https://github.com/MetaMask/metamask-extension/pull/45204) (closed unmerged 2026-08-13), [#45443 port Solana mocks to benchmarks](https://github.com/MetaMask/metamask-extension/pull/45443), [#45449 parallel mock suites drift silently](https://github.com/MetaMask/metamask-extension/issues/45449)
Contributor guide
Research direction
Start with the waitForTokenToBeDisplayed calls in test/e2e/benchmarks/flows/user-journey/onboarding-new-wallet.ts:163 and onboarding-import-wallet.ts:184, then inspect the benchmark mock routing in performance-mocks.ts and mock-responses.ts. Run the onboarding benchmark and instrument the wait, request log, retry count, and token presence before selecting the fix. Done means p(slow) is below 2%, the total metric is no longer bimodal, and the three metrics and gate assertions in test/e2e/benchmarks/utils/gated-metrics.ts are restored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ci-cd, performance, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100