MetaMask / MetaMask/metamask-extension

[P1] No stable real-world benchmark measure once live runs leave the per-commit path

Open
#45,472 1 comment 0 reactions 0 assignees View on GitHub
INVALID-ISSUE-TEMPLATE ta-triaged
Dominant language
TypeScript
Stars
13.2k
Forks
5.6k
Avg merge
2d 5h
Merged PRs (30d)
451

Description

**Parent Epic:** [#6944 Performance Quality Gates](https://github.com/MetaMask/MetaMask-planning/issues/6944) → [#7182 Benchmarks: Reliability](https://github.com/MetaMask/MetaMask-planning/issues/7182) → [#7181 Benchmark Harness Fixes](https://github.com/MetaMask/MetaMask-planning/issues/7181)
**File:** `.github/workflows/` (new scheduled workflow), `.github/workflows/run-benchmarks.yml`
**Size:** M | **Hours:** ~8-12

---

## Problem

How the live benchmark data is used should decide where it is collected. Four purposes were separated in the release thread, and they do not share an answer:

| purpose | best source |
|---|---|
| How the app performs for users | production telemetry — nothing in CI beats it |
| **A consistent real-world measure over time** | **scheduled CI, live calls, most recent release** |
| Drift between mock timings and reality | scheduled CI — [#45446 CDN cache state dominates live timings](https://github.com/MetaMask/metamask-extension/issues/45446) |
| Client-side regressions and the gate | mocks, strictly better — [#45455 mocked per-commit, live on a schedule](https://github.com/MetaMask/metamask-extension/pull/45455), [#45205 restore gate signal](https://github.com/MetaMask/metamask-extension/issues/45205) |

This ticket is the second row. [#45446 live-population benchmark timings are dominated by CDN cache state](https://github.com/MetaMask/metamask-extension/issues/45446) is the third.

Nothing covers the second row, and nothing covers the third either. The per-commit live path is gone: [#45679 use mocked requests for benchmarks on `main` and `release/*`](https://github.com/MetaMask/metamask-extension/pull/45679) merged 2026-08-24, and at `0350a018` [`shouldUseMockedRequests()`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/mock-config.ts#L12-L15) is a hardcoded `return true` for every ref. That PR was explicit that it stopped short of deleting the live-path code because a scheduled run might restore it, and named [#45455 gate per-commit runs on mocked, measure live on a schedule](https://github.com/MetaMask/metamask-extension/pull/45455) as the follow-up. [#45455 mocked per-commit, live on a schedule](https://github.com/MetaMask/metamask-extension/pull/45455) is open as of 2026-09-02, so the extension has no real-world benchmark measurement of any kind, and the `mock-config.ts` comment `TODO: Add a CI workflow that uses unmocked requests, or delete this dead code` is the open decision this ticket answers.

### Why the most recent release

A stable measure wants the confounders held still. Production moves with userbase, device mix and browser versions. `main` moves with whatever landed that hour, so one bad commit contaminates the series.

The most recent release is the only reference that is both real and stable, and it is what users are running. For this purpose it beats `main` and the release candidate, neither of which is affected by bugs that never shipped.

---

## Solution

A scheduled workflow, separate from the per-commit path:

1. **Run the suite live against the most recent release tag** on a fixed cadence — not `main`, not the RC. This requires restoring a way to disable mocks per run, since `shouldUseMockedRequests()` no longer branches.
2. **Publish to its own series**, keyed by population, so it never blends with the mocked per-commit series. [#45455 mocked per-commit, live on a schedule](https://github.com/MetaMask/metamask-extension/pull/45455) lands the `mockMode` split and the separate stats files; this consumes them.
3. **Gate nothing.** Output feeds trend detection and drift alarms; it blocks no merge.

Cadence and sample count are set from the first weeks of data, not guessed — [#45446 CDN cache state dominates live timings](https://github.com/MetaMask/metamask-extension/issues/45446)'s cache variance may mean one run per schedule is too few. Extending to the last N releases turns a moving point into a release-over-release comparison, and is deferred until the single-release version produces something.

---

## Acceptance Criteria

- [ ] A scheduled workflow runs the suite live against the most recent release tag
- [ ] Mocks can be disabled per run again, replacing the hardcoded `return true` in `shouldUseMockedRequests()` — or the dead live-path code is deleted and this ticket is closed as declined
- [ ] It publishes to a series distinct from the mocked per-commit series
- [ ] It blocks no merge and gates nothing
- [ ] Cadence and sample count are recorded with the reasoning that set them
- [ ] The series is queryable release-over-release
- [x] `main` and `release/*` no longer run live on the per-commit path — [#45679 mock `main` and `release/*` too](https://github.com/MetaMask/metamask-extension/pull/45679), merged 2026-08-24

---

## Labels

`team-extension-platform`, `area-CI`, `area-performance`

---

## Dependencies

**Depends on:** [#45455 gate per-commit runs on mocked, measure live on a schedule](https://github.com/MetaMask/metamask-extension/pull/45455) (population split, separate series files), [#45451 `main` benchmark baseline publishing stopped at GitHub's 100 MB file limit](https://github.com/MetaMask/metamask-extension/issues/45451) (series size cap — a third series otherwise hits the same limit)
**Related:** [#45446 live-population benchmark timings are dominated by CDN cache state](https://github.com/MetaMask/metamask-extension/issues/45446), [#45205 restore benchmark gate signal](https://github.com/MetaMask/metamask-extension/issues/45205), [#45204 stop gating the live-network benchmark population against mocked ceilings](https://github.com/MetaMask/metamask-extension/pull/45204) (closed unmerged 2026-08-13)

**Out of scope:** detecting *API* performance regressions — a scheduled job per release against UAT endpoints, alerting the API platform team, plus third-party API tracking. Different consumer, different endpoints, different alerting. It is not client performance work and should not be absorbed here without that team.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/run-benchmarks.yml and the new workflow location, then read test/e2e/benchmarks/utils/mock-config.ts and the changes planned in PR #45455. Run the existing benchmark workflow to understand its outputs and release-tag handling. Done means a scheduled live run targets the latest release, publishes a distinct queryable series, records cadence and sample reasoning, and gates no merge.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, performance, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.