MetaMask / MetaMask/metamask-extension
[P1] Gated benchmarks that never ran score as a pass
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 13.2k
- Forks
- 5.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 451
Description
**Parent Epic:** [#7588 Gate Verdicts and Output](https://github.com/MetaMask/MetaMask-planning/issues/7588)
**File:** `development/metamaskbot-build-announce/compare-benchmarks.ts`
**Size:** S | **Hours:** ~3-4
---
## Problem
A gated benchmark that never produced a result does not fail the quality gate. It is absent from the comparison set, and absence is scored as nothing rather than as a missing check, so the gate passes.
The gate's input is whatever the artifact directory happens to hold: `loadCurrentBenchmarks` calls `await fs.readdir(dirPath)` and iterates the `.json` files it finds ([`compare-benchmarks.ts:55-69`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/development/metamaskbot-build-announce/compare-benchmarks.ts#L55-L69)). No declared set is compared against that listing, so a benchmark that produced nothing shortens the list rather than raising a signal.
This is the failure mode a gate must not have. Every other way of not-passing is loud: a threshold breach reports, a crashed job reports. A benchmark that silently did not run reports a green check that means "nothing was measured", and the reader cannot distinguish it from "everything measured fine".
It compounds with the two adjacent holes:
- [**#45460**](https://github.com/MetaMask/metamask-extension/pull/45460) — a PR that changes only benchmark-owned code (harness, mocks, thresholds, the gate allowlist) matches its base build hash, reuses that build, and resolves `needs-benchmarks: false`, so the change goes unmeasured and the gate passes.
- [**#45451**](https://github.com/MetaMask/metamask-extension/issues/45451) — `main` baseline publishing has been rejected by the remote since 2026-07-28 because `stats/main/performance_data.json` reached GitHub's 100 MiB blob limit, and `continue-on-error: true` on both the job and its commit step reports the push failure as success. The historical half of every comparison since is frozen and also passes.
All three share one shape: **a missing input scores as a pass.** Fixing them individually is right, but the class is worth naming — any gate input that can go absent needs an explicit absent-is-failure rule, not a default.
---
## Solution
Covered by [#45047 (a no-result benchmark blocks when it owns a gated metric)](https://github.com/MetaMask/metamask-extension/pull/45047): treat a gated metric with no result as a gate failure rather than an omission, and report which metric was expected and did not arrive.
Beyond that PR, the class-level work:
1. **An expected-set check.** The gate imports `GATED_METRICS`, the runtime set exported from `test/e2e/benchmarks/utils/gated-metrics.ts`; it should assert that every entry either produced a result or is explicitly waived, and fail naming the difference.
2. **Distinguish absent from waived.** A metric demoted on purpose — as `onboardingNewWallet.total`, `onboardingNewWallet.doneButtonToAssetList` and `onboardingImportWallet.doneButtonToHomeScreen` were in [#45444 (demote the three bimodal onboarding metrics out of the gate)](https://github.com/MetaMask/metamask-extension/pull/45444) — and a metric that vanished because its job died must not look alike in the output.
---
## Acceptance Criteria
- [ ] A run in which a gated benchmark produces no result fails the quality gate
- [ ] The failure names the metric that was expected and did not arrive
- [ ] A deliberately non-gated metric is distinguishable in the output from one that went missing
- [ ] A test covers the missing-result path and fails without the change
---
## Labels
`team-extension-platform`, `area-CI`, `area-testSuite`
---
## Dependencies
**Addressed by:** #45047 (a no-result benchmark blocks when it owns a gated metric)
**Related:** #45460 (a benchmark-only change reuses its base build and is never measured), #45451 (a frozen `main` baseline also passes), #45046 (the standing `main` red that surfaced this)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read development/metamaskbot-build-announce/compare-benchmarks.ts, especially loadCurrentBenchmarks, and compare its discovered files with GATED_METRICS from test/e2e/benchmarks/utils/gated-metrics.ts. Check the existing coverage around benchmark comparison, then verify that a missing gated result names the metric, distinguishes deliberate non-gated metrics, and fails the quality gate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100