MetaMask / MetaMask/metamask-extension
[P1] Gate output cannot tell a developer whether a red is about their change
- Dominant language
- TypeScript
- Stars
- 13.2k
- Forks
- 5.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 451
Description
**Parent Epic:** [#7588 Gate Verdicts and Output](https://github.com/MetaMask/MetaMask-planning/issues/7588)
**File:** `development/metamaskbot-build-announce/compare-benchmarks.ts`, `test/e2e/benchmarks/utils/statistics.ts`
**Size:** M | **Hours:** 4-6
---
## Problem
The quality gate's output does not carry enough information for a developer to decide whether a red result is about their change. That is what drives reaching for `skip-benchmark-gate` on PRs that have nothing to do with benchmarks.
Measured over **87 completed `main`-branch push runs**, 7 days to `2026-09-01T12:07:29Z` (workflow `60751188`, `event=push`, `status=completed`):
| observation | count |
|---|---|
| Runs printing `RESULT: PASS` that **also** print `absolute ceiling exceeded` | **70 of 70** |
| Benchmarks amber in **every** run | 9, across 85 of 85 |
| Runs whose gate ran at all | 85 of 87 |
Not one clean pass occurred in the window. Three separate gaps produce this, each verified against `main` at `0350a0187`:
**1. The tier is not disclosed.** The line reads `p75: 11098ms (absolute ceiling exceeded, limit 11050ms)`, built at [`compare-benchmarks.ts:262-266`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/development/metamaskbot-build-announce/compare-benchmarks.ts#L262-L266) from `violation.value` and `violation.threshold` alone. Whether `11050` is the warn ceiling or the fail ceiling is not stated, so a saturated warn tier and a genuine fail render identically. `severity` reaches the output only as an icon, through [`violationIcon`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/development/metamaskbot-build-announce/compare-benchmarks.ts#L149-L153).
**2. The widening is not disclosed.** `ThresholdViolation.threshold` is the fully-adjusted value: [`getEffectiveThreshold`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/statistics.ts#L450-L464) multiplies the base ceiling by the CI multiplier and then by the CV adjustment, and the violation records `cvAdjustment` beside it at [`statistics.ts:494-511`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/statistics.ts#L494-L511). That field is never read by the gate's printer — `cvAdjustment` appears nowhere in `compare-benchmarks.ts`, where `severity` appears nine times. So a ceiling widened by this run's own variance prints the same as a base ceiling, and the same measured value can print as within-limit in one run and over-limit in another with no visible difference. Observed: `11108ms` has both passed and failed against a printed `limit 11050ms`.
**3. Some exclusions emit no line at all.** `statistics.ts` drops metrics at three points, each a bare `continue`:
- no threshold configured for the metric — [`statistics.ts:587-589`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/statistics.ts#L587-L589)
- `stats.dataQuality === 'unreliable'` — [`statistics.ts:592-594`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/statistics.ts#L592-L594)
- `cv > CV_THRESHOLDS.POOR` — [`statistics.ts:637-639`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/statistics.ts#L637-L639)
So the count of metrics excluded from a given verdict is not recoverable from CI output. The module's only three `console.log` calls sit in [`logThresholdResult`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/statistics.ts#L814-L827), which prints violations and is called only from `test/e2e/benchmarks/run-benchmark.ts` — so console output exists in this module, and none of it reports an exclusion.
## Solution
1. **Name the tier and the widening on the violation line.** `ThresholdViolation` already carries `severity` and `cvAdjustment`; neither reaches the printed text. — [#45933 (name the tier and CV widening on a ceiling violation)](https://github.com/MetaMask/metamask-extension/pull/45933)
2. **Log the three silent exclusions in `statistics.ts`**, with the metric id and the reason. This is also the unmet acceptance criterion of [MetaMask/MetaMask-planning#7480 (harden the absolute benchmark gate against within-noise breaches)](https://github.com/MetaMask/MetaMask-planning/issues/7480), closed 2026-07-22: its AC reads "Gated metrics whose components exceed the CV cap are skipped **and logged** (not failed), CV cap a single named constant".
3. **Emit a per-run coverage line** — metrics evaluated over metrics expected — so a reader can see the denominator without reconstructing it.
## Acceptance Criteria
- [ ] A violation line states whether the ceiling is the warn or the fail tier
- [ ] A widened ceiling states the factor and the pre-widening value
- [ ] Every metric excluded from scoring produces exactly one log line naming the metric and the reason
- [ ] The gate prints the count of metrics evaluated against the count expected
- [ ] A reader can determine, from CI output alone, whether a red is attributable to the PR under test
## Labels
`team-extension-platform`
## Dependencies
**Addressed by:** #45933 (name the tier and CV widening on a ceiling violation) — covers item 1 only
**Related:** MetaMask/metamask-extension#45544 (the `CV_ADAPTIVE_MAX` cliff — widening applied *wrongly* above CV 50; this ticket is about it not being *disclosed*), MetaMask/MetaMask-planning#7480 (closed 2026-07-22 with the logging AC unmet)
Contributor guide
Research direction
Read development/metamaskbot-build-announce/compare-benchmarks.ts and test/e2e/benchmarks/utils/statistics.ts, starting with the existing violation printer and the three exclusion branches. Check #45933 before working, since it covers tier and CV widening; the remaining work is to log exclusions, print evaluated-versus-expected coverage, and satisfy the CI-output acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ci-cd, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100