MetaMask / MetaMask/metamask-extension

[P0] `main` benchmark baseline publishing silently stopped at GitHub's 100 MB file size limit

Open
#45,451 0 comments 0 reactions 1 assignee Claimed by @HowardBraham View on GitHub
area-CI INVALID-ISSUE-TEMPLATE ta-bot-triage team-extension-platform
Dominant language
TypeScript
Stars
13.2k
Forks
5.6k
Avg merge
2d 5h
Merged PRs (30d)
451

Description

**Parent Epic:** [#6944 Performance Quality Gates](https://github.com/MetaMask/MetaMask-planning/issues/6944)
**Sub-epic:** [#7587 Baseline Publishing](https://github.com/MetaMask/MetaMask-planning/issues/7587)
**Stream:** [#7183 Announcements](https://github.com/MetaMask/MetaMask-planning/issues/7183) — closed 2026-09-02 as completed.
**File:** `.github/scripts/benchmark-stats-commit.sh`; `.github/workflows/run-benchmarks.yml`, `store-benchmark-stats` job at [line 322](https://github.com/MetaMask/metamask-extension/blob/199772f9d93b95487f9606fbe1a873b9f8d63dc0/.github/workflows/run-benchmarks.yml#L322), commit step at [line 365](https://github.com/MetaMask/metamask-extension/blob/199772f9d93b95487f9606fbe1a873b9f8d63dc0/.github/workflows/run-benchmarks.yml#L365) (sha `199772f9d93b`, read 2026-09-02).
**Size:** M | **Hours:** ~6-10

---

## Problem

`stats/main/performance_data.json` in `MetaMask/extension_benchmark_stats` has grown past **GitHub's hard 100 MiB file size limit**.
Every push of a new commit's data is rejected by the remote, so the `main` baseline has been **frozen since 2026-07-28T09:05:23Z** — 36 days as of 2026-09-02 — and it cannot recover on its own, because the file only grows.
The job reports **success** while this happens, because both the job (`store-benchmark-stats`, line 322) and its commit step (line 365) carry `continue-on-error: true`.

A fix was proposed and closed without merging: [metamask-extension#45445](https://github.com/MetaMask/metamask-extension/pull/45445) implemented a keep-last-100 retention cap via a `RETAINED_COMMITS` variable, verified against a synthetic 120-entry series, and was closed 2026-09-01 with no comment on the PR stating why.
The current script ([sha `d74fdbbd4851`](https://github.com/MetaMask/metamask-extension/blob/d74fdbbd485130424e1727ef6a752f0ccb14c3dd/.github/scripts/benchmark-stats-commit.sh)) has no trim, prune, retain, slice, truncate or limit of any kind — the retention fix never landed, and the baseline remains unbounded and frozen.

### Observation

Still failing today. From the `store-benchmark-stats` job in [run 33635300699](https://github.com/MetaMask/metamask-extension/actions/runs/33635300699/job/100272421919) (push to `main` at `0350a01878e0`, 2026-09-02T13:46:11Z), which the API reports with `conclusion: success`:

```
[main 6118870] Adding performance benchmark data for main at commit: 0350a01878e0d5e0eb1553b18ca7d653b4157cbb
1 file changed, 7460 insertions(+)
remote: error: File stats/main/performance_data.json is 100.15 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/MetaMask/extension_benchmark_stats'
##[error]Process completed with exit code 1.
```

The original observation, from run 31110781089 on 2026-08-06, showed the same rejection nine days into the freeze.
The run above shows it unchanged 36 days in.

Corroborating facts, re-verified 2026-09-02:

| fact | value |
|---|---|
| current file size (`main`, remote) | 104,772,933 bytes = **99.92 MiB**, unchanged since the freeze |
| GitHub hard limit | 104,857,600 bytes = 100.00 MiB |
| bytes added per publish attempt | ~180,000–270,000 (three consecutive commits, 2026-07-27/28) |
| lines added per publish attempt | 5,332 (2026-08-06 run) to 7,460 (2026-09-02 run) |
| last successful `main` write | **2026-07-28T09:05:23Z** |
| last write to the repo (any path) | 2026-09-01T15:15:06Z — `release/13.47.0` |
| commits accumulated in the frozen file | **822**, confirmed by downloading the file and counting keys |
| days frozen | **36**, as of 2026-09-02 |

`release/*` still publishes: `stats/release-13.47.0/performance_data.json` was written 2026-09-01T15:15:06Z and is 1,264,595 bytes (1.21 MiB), far under the limit, because each release branch gets its own file.
That is why the repository's commit feed looks healthy.

### Why this matters

- **Every PR comment's historical delta is computed against a frozen baseline**, stale by 36 days as of 2026-09-02 and growing. The percentages are not wrong-looking; they are quietly measuring against an old world.
- **It is permanent without intervention.** The file is already over the limit; no future push can succeed.
- **It is invisible.** The job is green, so nothing surfaces it.
- **It blocks two items already in this epic:** [#7287 Self-Calibration](https://github.com/MetaMask/MetaMask-planning/issues/7287) reads this history, and the mocked-series work in [extension#45205](https://github.com/MetaMask/metamask-extension/issues/45205) would hit the same wall as soon as its file grows.

This is **not** a recurrence of [planning#7279 jq ARG_MAX exit 126](https://github.com/MetaMask/MetaMask-planning/issues/7279) (closed 2026-06-09, fixed by passing the payload via stdin instead of argv).
That fix is intact: the current script reads `presets_json` via stdin at [line 137](https://github.com/MetaMask/metamask-extension/blob/d74fdbbd485130424e1727ef6a752f0ccb14c3dd/.github/scripts/benchmark-stats-commit.sh#L137), not on argv.
This is a different failure mode — unbounded file growth into a platform limit — and it recurs for the same underlying reason the earlier fix didn't touch: `benchmark-stats-commit.sh` has never had a retention rule.

---

## Solution

1. **Stop the silent failure.** Remove `continue-on-error: true` from the commit step, or have the script exit non-zero and surface a distinguishable annotation. A publishing failure must not report success — that property is what let this run unnoticed for 36 days and counting.
2. **Bound the file.** Options, roughly in order of preference:
- **Rotate by period** — `stats/main/performance_data-{YYYY-QN}.json`, with the consumer reading the most recent N files. Preserves history, bounded per file.
- **Prune on write** — retain a rolling window (the baseline aggregation only reads the latest 5 commits; `historical-comparison.ts` slices to 5). Most of the 822 commits are never read.
- **Split per preset** — smaller files, and consumers already fetch by preset.
3. **Unblock the current file**, which is already over the limit and cannot be pushed to at all. Rotation naturally does this by starting a new file.
4. **Add a staleness alarm** — if the newest entry is older than N days, fail loudly. This is the generic guard that would have caught both this and [planning#7279 jq ARG_MAX exit 126](https://github.com/MetaMask/MetaMask-planning/issues/7279).

A closed, unmerged PR already worked out one version of (2): [metamask-extension#45445](https://github.com/MetaMask/metamask-extension/pull/45445) added a keep-last-100 cap via a `RETAINED_COMMITS` variable and verified the retention expression against a synthetic series before any real cutover.
Whoever picks this up should read it before starting from zero; nothing in the PR or its closure explains why it wasn't merged.

---

## Acceptance Criteria

- [ ] A publishing failure fails the job visibly; it can no longer report success
- [ ] `stats/main/performance_data.json` (or its successor) is bounded and pushes succeed
- [ ] `main` baseline resumes updating, verified over ≥3 consecutive `main` pushes
- [ ] Consumers (`historical-comparison.ts`, `performance-benchmarks.ts`) read the rotated/pruned layout correctly
- [ ] A staleness alarm fires when the newest baseline entry exceeds an agreed age
- [ ] The same bound is applied to `release/*` files before they reach the limit

---

## Labels

`team-extension-platform`, `area-CI`

---

## Dependencies

**Blocks:** [#7287 Self-Calibration](https://github.com/MetaMask/MetaMask-planning/issues/7287), [extension#45205 Restore benchmark gate signal](https://github.com/MetaMask/metamask-extension/issues/45205)
**Related:** [planning#7279 jq ARG_MAX exit 126](https://github.com/MetaMask/MetaMask-planning/issues/7279) (different failure mode, same symptom; closed 2026-06-09), [#7183 Announcements](https://github.com/MetaMask/MetaMask-planning/issues/7183) (closed 2026-09-02)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.