MetaMask / MetaMask/metamask-extension
[P1] startupPowerUserHome exits 0 with no measurements, and the gate scores 24 of 25
- Dominant language
- TypeScript
- Stars
- 13.2k
- Forks
- 5.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 451
Description
**Parent Epic:** [#6944 Performance Quality Gates](https://github.com/MetaMask/MetaMask-planning/issues/6944) → [#7182 Benchmarks: Reliability](https://github.com/MetaMask/MetaMask-planning/issues/7182) → [#7181 Benchmark Harness Fixes](https://github.com/MetaMask/MetaMask-planning/issues/7181)
**File:** `test/e2e/benchmarks/flows/startup/power-user-home.ts`
**Size:** M | **Hours:** 4-8
---
## Problem
`chrome-webpack-startupPowerUserHome` exhausts its retries, writes an error object instead of measurements, **exits 0**, and reports success. The gate then drops the benchmark from scoring and prints `PASS` over a denominator of 24 instead of 25.
The artifact it uploads, against its own passing firefox sibling from the same run:
```json
// benchmark-chrome-webpack-startupPowerUserHome.json — 288 bytes
{ "startupPowerUserHome": { "error": "Error: Retry limit reached" } }
```
```json
// benchmark-firefox-webpack-startupPowerUserHome.json — 2,811 bytes
{ "startupPowerUserHome": { "testTitle": "measurePagePowerUser", "persona": "powerUser",
"mean": { "uiStartup": 2823.16, "load": 1816.01, "domContentLoaded": 1814.84, ... } } }
```
### This is deterministic, not variance
Four runs on the test fork, all at the same upstream commit `804a38ddfd98`, nothing varying but the run:
| run | legs green | gate output |
|---|---|---|
| [33509082641](https://github.com/consensys-test/metamask-extension-test-majorlift/actions/runs/33509082641) | 17 / 17 | `Total: 24 benchmarks` · `RESULT: PASS` |
| [33512902168](https://github.com/consensys-test/metamask-extension-test-majorlift/actions/runs/33512902168) | 17 / 17 | `Total: 24 benchmarks` · `RESULT: PASS` |
| [33516999236](https://github.com/consensys-test/metamask-extension-test-majorlift/actions/runs/33516999236) | 17 / 17 | `Total: 24 benchmarks` · `RESULT: PASS` |
| [33517022676](https://github.com/consensys-test/metamask-extension-test-majorlift/actions/runs/33517022676) | 17 / 17 | `Total: 24 benchmarks` · `RESULT: PASS` |
Identical 4 of 4. **No variance-reduction work will move this** — it is not flake. It also reproduces on demand, which makes it cheap to diagnose relative to anything measured across commits.
Upstream it appears in **76 of 85** `main` runs in the 7 days to `2026-09-01T12:07:29Z`. That spread is commit-to-commit variation, not run-to-run.
### Why exit 0
The swallow is three hops, all readable at `0350a018`:
1. `power-user-home.ts` delegates to `runPageLoadBenchmark`, which calls `retry()` at [`runner.ts:451`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/runner.ts#L451) with no local try/catch, so retry exhaustion throws. The thrown message originates at [`development/lib/retry.js:26`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/development/lib/retry.js#L26); the string `Retry limit reached` appears nowhere under `test/e2e/benchmarks/`.
2. [`run-benchmark.ts:358-373`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/run-benchmark.ts#L358-L373) wraps each benchmark file in a try/catch that logs the error and records `allResults[resultKey] = { error: String(error) }`. The only exit-code path is `main().catch(exitWithError)`, which this catch prevents from firing — so the process exits 0 with an error object in its output.
3. [`compare-benchmarks.ts:96-101`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/development/metamaskbot-build-announce/compare-benchmarks.ts#L96-L101) skips any entry missing `p75`/`p95` — the exact shape of an `{ error }` entry — with `Skipping "": missing p75/p95 (benchmark likely failed).` and `continue`. The entry never enters `comparisons`, so it is not counted as a failure and it silently shrinks the denominator printed at [`compare-benchmarks.ts:386`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/development/metamaskbot-build-announce/compare-benchmarks.ts#L386).
### Proximate cause of the retry exhaustion
[Job 99808836384](https://github.com/MetaMask/metamask-extension/actions/runs/33492463412/job/99808836384): `getSubscriptions` errors against `subscription.api.cx.metamask.io` → `TimeoutError` waiting on `.controller-loaded` → `Retry limit reached`.
**Whether that is a network dependency, a fixture problem, or a real product regression is not determined here.** The other 75 occurrences were counted by the aggregate skip line, not read individually, so the proximate cause is not claimed identical in each.
Note that every benchmark run is mocked as of [#45679 use mocked requests for benchmarks on `main` and `release/*`](https://github.com/MetaMask/metamask-extension/pull/45679) (merged 2026-08-24), so a live `subscription.api.cx.metamask.io` call inside a benchmark is itself a mock-coverage gap, not expected behavior.
## Why existing tickets do not cover this
- [#45450 gated benchmarks that never ran score as a pass](https://github.com/MetaMask/metamask-extension/issues/45450) covers the gate *passing* over a missing measurement. It does not cover why the measurement is missing.
- [#45669 a timed-out benchmark blocks CI as an opaque cancellation](https://github.com/MetaMask/metamask-extension/issues/45669) covers a shard that times out and is cancelled. This one completes, succeeds, and uploads.
## Acceptance Criteria
- [ ] The cause of `Retry limit reached` on this flow is identified and named
- [ ] A run of this benchmark either produces `p75`/`p95`, or fails its own step
- [ ] The four-run pinned-tree check produces `Total: 25 benchmarks` with zero `Skipping` lines
- [ ] If the cause is a live network dependency, it is mocked or the metric is de-gated with that stated
## Labels
`team-extension-platform`, `area-CI`
## Dependencies
**Related:** [#45450 gated benchmarks that never ran score as a pass](https://github.com/MetaMask/metamask-extension/issues/45450), [#45934 gate output cannot tell a developer whether a red is about their change](https://github.com/MetaMask/metamask-extension/issues/45934)
Contributor guide
Research direction
Start with test/e2e/benchmarks/flows/startup/power-user-home.ts and trace runPageLoadBenchmark through test/e2e/benchmarks/utils/runner.ts:451. Then inspect run-benchmark.ts:358-373 and development/metamaskbot-build-announce/compare-benchmarks.ts:96-101, and run the pinned-tree benchmark check. Done means the cause is identified, the benchmark produces p75/p95 or fails its step, and the gate reports all 25 benchmarks without skipped entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100