MetaMask / MetaMask/metamask-extension
[P1] Benchmark runs are not observable: iteration failures emit nothing, re-runs overwrite their own data
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 13.2k
- Forks
- 5.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 451
Description
**Parent Epic:** [#6944 Performance Quality Gates](https://github.com/MetaMask/MetaMask-planning/issues/6944) → [#7182 Benchmarks: Reliability](https://github.com/MetaMask/MetaMask-planning/issues/7182) → [#7206 CI Reliability Improvements](https://github.com/MetaMask/MetaMask-planning/issues/7206)
**File:** `test/e2e/benchmarks/utils/runner.ts`, `test/e2e/benchmarks/send-to-sentry.ts`, `.github/workflows/run-benchmarks.yml`
**Size:** S | **Hours:** ~4-6
---
## Problem
The benchmark pipeline cannot answer questions about its own runs. Three gaps compound, all cheap to close. Line references are pinned to `555c1467fa78b04028c338dbc9f7466144a7ec1e`; all three are confirmed still present on `main` (read 2026-09-02).
### 1. A failed or stalled iteration emits nothing
[`runner.ts`](https://github.com/MetaMask/metamask-extension/blob/555c1467fa78b04028c338dbc9f7466144a7ec1e/test/e2e/benchmarks/utils/runner.ts#L153) reports only on success:
```ts
for (let i = 0; i < iterations; i++) {
const result = await runWithRetries(benchmarkFn, retries);
allResults.push(result);
if (result.success) {
...
performanceTracker.generateReport(`${name} - iteration ${i + 1}`, name);
} else {
failedRuns += 1; // silent
}
}
```
So a run that stalled, a run whose iterations all failed, and a run that stopped after the first iteration are identical in a CI log. This blocked attribution on `chrome-webpack-userJourneyTransactions`: run 31423339197 (2026-08-10) shows one iteration's report at 19:25:26 and then 31 minutes of nothing before cancellation. Whether iteration 2 began cannot be recovered.
`runBenchmarkWithIterations` also has no unit coverage.
An open PR, #45432, fixes this specific gap — see Dependencies.
### 2. Re-runs overwrite their own S3 data
Both upload sites key on `github.run_id`, which is stable across attempts, and `run_attempt` appears nowhere in the workflow — confirmed still true on `main` at both upload sites (`.github/workflows/run-benchmarks.yml`, 2026-09-02):
```
${AWS_S3_BUCKET}/${repo}/${run_id}/benchmarks/benchmark-{browser}-{buildType}-{pageType}.json
```
Attempt 2 writes to the same key as attempt 1. The loss is biased rather than uniform: the runs most likely to be re-run are the ones that failed, so the surviving object is disproportionately the passing attempt and the failing measurement is the one destroyed. On a PR at attempt 4, three attempts of benchmark data no longer exist.
### 3. Sentry has no run or attempt identifier
[`send-to-sentry.ts:207`](https://github.com/MetaMask/metamask-extension/blob/555c1467fa78b04028c338dbc9f7466144a7ec1e/test/e2e/benchmarks/send-to-sentry.ts#L207) sets `ci.branch`, `ci.prNumber`, `ci.commitHash`, `ci.browser`, `ci.buildType` — and nothing identifying the run or attempt. Confirmed unchanged in `baseCiAttributes` on `main`, 2026-09-02.
The two stores end up with inverted affordances. S3 is addressable by run and cannot be searched by commit or branch. Sentry is queryable by commit and branch and cannot separate runs. Neither answers "show me every attempt on this commit", and after (2) the data for the earlier attempts is gone regardless.
---
## Solution
1. Emit an iteration-start line before each iteration, and a failure line carrying `result.error` in the `else` branch. Add unit coverage for the loop including a stalled-iteration case.
2. Add `${{ github.run_attempt }}` to both S3 key templates.
3. Add `ci.runId` and `ci.runAttempt` to `baseCiAttributes`, from `GITHUB_RUN_ID` and `GITHUB_RUN_ATTEMPT`.
(2) and (3) together make the two stores joinable to each other and to GitHub.
---
## Acceptance Criteria
- [ ] Every iteration announces itself before running, and a failed iteration logs its error rather than being dropped
- [ ] A stalled run names the iteration that stalled
- [ ] Tests for the iteration loop fail without the logging change and pass with it
- [ ] Re-running a workflow no longer overwrites the previous attempt's S3 objects
- [ ] A Sentry query can isolate a single run and a single attempt
- [ ] Given a commit, every attempt's data is retrievable from at least one store
---
## Labels
`team-extension-platform`, `area-testSuite`, `area-CI`
---
## Dependencies
**Blocks:** attribution of the `userJourneyTransactions` stall (#45266, #45205)
**Related:** #45352 (open, unmerged — fixes a stale build-listing causing redundant re-runs; a re-run that then finds a matching build hash and skips is the intended outcome per #45352's own current description, not a defect, but item 2 above still applies to it: whichever attempt's S3 upload runs last is the only one whose data survives)
**In progress:** #45432 (open, unmerged) fixes item 1 only — its own description states items 2 and 3 are "deliberately left out to keep this reviewable" and lists them as follow-up work. If #45432 merges carrying its current `Fixes: #45431` link, this issue will auto-close with items 2 and 3 still unaddressed; whoever merges it should re-open or re-file for the remainder.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with test/e2e/benchmarks/utils/runner.ts and its runBenchmarkWithIterations loop, then inspect test/e2e/benchmarks/send-to-sentry.ts and .github/workflows/run-benchmarks.yml. Add or run unit coverage for iteration logging, and verify the workflow keys and Sentry attributes distinguish run attempts. Done means failed or stalled iterations are identifiable and rerun data remains separately retrievable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, github-actions, typescript
- Domain
- ci-cd, observability, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100