MetaMask / MetaMask/metamask-extension

[P1] Gated benchmarks that never ran score as a pass

Open
#45,450 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

INVALID-ISSUE-TEMPLATE ta-ai-fixable ta-needs-engineer-escalation ta-triaged team-extension-platform
Dominant language
TypeScript
Stars
13.2k
Forks
5.6k
Avg merge
2d 5h
Merged PRs (30d)
451

Description

**Parent Epic:** [#7588 Gate Verdicts and Output](https://github.com/MetaMask/MetaMask-planning/issues/7588)
**File:** `development/metamaskbot-build-announce/compare-benchmarks.ts`
**Size:** S | **Hours:** ~3-4

---

## Problem

A gated benchmark that never produced a result does not fail the quality gate. It is absent from the comparison set, and absence is scored as nothing rather than as a missing check, so the gate passes.

The gate's input is whatever the artifact directory happens to hold: `loadCurrentBenchmarks` calls `await fs.readdir(dirPath)` and iterates the `.json` files it finds ([`compare-benchmarks.ts:55-69`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/development/metamaskbot-build-announce/compare-benchmarks.ts#L55-L69)). No declared set is compared against that listing, so a benchmark that produced nothing shortens the list rather than raising a signal.

This is the failure mode a gate must not have. Every other way of not-passing is loud: a threshold breach reports, a crashed job reports. A benchmark that silently did not run reports a green check that means "nothing was measured", and the reader cannot distinguish it from "everything measured fine".

It compounds with the two adjacent holes:

- [**#45460**](https://github.com/MetaMask/metamask-extension/pull/45460) — a PR that changes only benchmark-owned code (harness, mocks, thresholds, the gate allowlist) matches its base build hash, reuses that build, and resolves `needs-benchmarks: false`, so the change goes unmeasured and the gate passes.
- [**#45451**](https://github.com/MetaMask/metamask-extension/issues/45451) — `main` baseline publishing has been rejected by the remote since 2026-07-28 because `stats/main/performance_data.json` reached GitHub's 100 MiB blob limit, and `continue-on-error: true` on both the job and its commit step reports the push failure as success. The historical half of every comparison since is frozen and also passes.

All three share one shape: **a missing input scores as a pass.** Fixing them individually is right, but the class is worth naming — any gate input that can go absent needs an explicit absent-is-failure rule, not a default.

---

## Solution

Covered by [#45047 (a no-result benchmark blocks when it owns a gated metric)](https://github.com/MetaMask/metamask-extension/pull/45047): treat a gated metric with no result as a gate failure rather than an omission, and report which metric was expected and did not arrive.

Beyond that PR, the class-level work:

1. **An expected-set check.** The gate imports `GATED_METRICS`, the runtime set exported from `test/e2e/benchmarks/utils/gated-metrics.ts`; it should assert that every entry either produced a result or is explicitly waived, and fail naming the difference.
2. **Distinguish absent from waived.** A metric demoted on purpose — as `onboardingNewWallet.total`, `onboardingNewWallet.doneButtonToAssetList` and `onboardingImportWallet.doneButtonToHomeScreen` were in [#45444 (demote the three bimodal onboarding metrics out of the gate)](https://github.com/MetaMask/metamask-extension/pull/45444) — and a metric that vanished because its job died must not look alike in the output.

---

## Acceptance Criteria

- [ ] A run in which a gated benchmark produces no result fails the quality gate
- [ ] The failure names the metric that was expected and did not arrive
- [ ] A deliberately non-gated metric is distinguishable in the output from one that went missing
- [ ] A test covers the missing-result path and fails without the change

---

## Labels

`team-extension-platform`, `area-CI`, `area-testSuite`

---

## Dependencies

**Addressed by:** #45047 (a no-result benchmark blocks when it owns a gated metric)
**Related:** #45460 (a benchmark-only change reuses its base build and is never measured), #45451 (a frozen `main` baseline also passes), #45046 (the standing `main` red that surfaced this)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read development/metamaskbot-build-announce/compare-benchmarks.ts, especially loadCurrentBenchmarks, and compare its discovered files with GATED_METRICS from test/e2e/benchmarks/utils/gated-metrics.ts. Check the existing coverage around benchmark comparison, then verify that a missing gated result names the metric, distinguishes deliberate non-gated metrics, and fails the quality gate.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.