[P1] Live-population benchmark timings are dominated by CDN cache state

**Parent Epic:** [#6944 Performance Quality Gates](https://github.com/MetaMask/MetaMask-planning/issues/6944) → [#7182 Benchmarks: Reliability](https://github.com/MetaMask/MetaMask-planning/issues/7182) → [#7181 Benchmark Harness Fixes](https://github.com/MetaMask/MetaMask-planning/issues/7181)
**File:** `test/e2e/benchmarks/utils/mock-config.ts`, `test/e2e/benchmarks/mocks/performance-mocks.ts`, `.github/workflows/run-benchmarks.yml`
**Size:** M | **Hours:** ~8-12

---

## Problem

A benchmark run against live endpoints records a response time that is largely a property of CDN cache state rather than of anything in the commit. That makes a live population unusable as a per-commit gate, and it makes ceilings re-derived from one drift with cache behavior rather than with the product.

Two inputs, one measured here and one reported by API platform:

- **Reported.** API platform has said our live-endpoint calls mostly hit cache. That makes any live population a *mixture* of cached and origin-served responses, with the mixing ratio set by traffic from everyone else rather than by the run.
- **Measured.** `tokens.api.cx.metamask.io/v3/assets` serves `cache-control: max-age=10800` behind Cloudflare, and cache state moves the response time by ~2.8× on an otherwise identical request.

### Where the live population stands

`main` and `release/*` ran benchmarks against live endpoints until [#45679 use mocked requests for benchmarks on `main` and `release/*`](https://github.com/MetaMask/metamask-extension/pull/45679), merged 2026-08-24 (`784bdef`). At `0350a018`, [`shouldUseMockedRequests()`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/utils/mock-config.ts#L12-L15) is a hardcoded `return true` carrying the comment `TODO: Add a CI workflow that uses unmocked requests, or delete this dead code.` The branch-dependent form this ticket was written against is at [`mock-config.ts:33` on `fb38a10e`](https://github.com/MetaMask/metamask-extension/blob/fb38a10ed6949beb4884548d86cf7aa028dce5bc/test/e2e/benchmarks/utils/mock-config.ts#L33).

So there is no live population on the per-commit path to fix, and none anywhere else yet. Everything below applies to the scheduled live suite that [#45455 gate per-commit runs on mocked, measure live on a schedule](https://github.com/MetaMask/metamask-extension/pull/45455) and [#45472 no stable real-world benchmark measure once live runs leave the per-commit path](https://github.com/MetaMask/metamask-extension/issues/45472) would create. Without one, the cache measurement below describes a population nobody collects.

### Measurement

20 requests to `https://tokens.api.cx.metamask.io/v3/assets?assetIds=eip155:1/slip44:60`, 2026-08-11T21:30Z. Group A repeats the same URL (warm); group B appends a random `cb=` parameter each time to force an origin fetch. Full capture, including every header: [`tokens-api-cache-probe.txt`](https://majorlift-artifacts-share.s3.us-west-1.amazonaws.com/public/metamask/benchmarks/2026-08-11-tokens-api-cache/tokens-api-cache-probe.txt)

| `cf-cache-status` | n | median TTFB | mean | min | max |
|---|---:|---:|---:|---:|---:|
| `HIT` | 12 | **105ms** | 118ms | 68ms | 241ms |
| `MISS` | 8 | **292ms** | 283ms | 225ms | 328ms |

Ratio of medians **2.77×**. The response headers confirm the TTL directly:

```
cache-control: max-age=10800
age: 26
cf-cache-status: HIT
server: cloudflare
cf-ray: a29a5851791a0c60-SJC
```

**Scope of this capture.** It was taken from one vantage point on a developer machine, not from a CI runner. Within that scope it shows cache state alone moving this endpoint's response time by ~190ms at the median, with the two groups nearly disjoint — max `HIT` 241ms against min `MISS` 225ms, so they touch at the extremes and this is a shift in distribution rather than a clean threshold.

Two things it cannot speak to, both readable off the artifact itself:

- **Cross-edge spread.** Every `cf-ray` in the capture ends `-SJC`, so the file contains one edge and nothing about the variation between edges. That part of the problem statement is an inference from CDN topology.
- **Call volume.** Nothing here counts how many such requests a benchmark flow issues, so the per-request delta cannot yet be turned into a per-flow one.

### The three ways caching enters the measurement

Enumerated in [planning#7297 backend caching distorts real-network benchmark measurements](https://github.com/MetaMask/MetaMask-planning/issues/7297) before the real-network mode existed; the measurement above confirms the first.

1. **Warm-path CDN responses.** The first iteration's TTFB includes origin-fetch plus CDN populate; later iterations skip both. Inter-iteration variance collapses artificially low while cold-path regressions stay masked.
2. **App-server LRU caches.** Token-list, feature-flag and price endpoints hold in-memory caches keyed on recent requests. A benchmark runner issuing repeated identical requests warms them and never measures the cold path.
3. **Cross-iteration cache reuse.** Two runs scheduled close together share partially-warmed CDN state, so a rolling baseline trends downward because the hit rate improved rather than because the app got faster.

Net effect: a live baseline trusts numbers no real user ever sees.

### A discriminator, and one metric it already clears

[planning#7297](https://github.com/MetaMask/MetaMask-planning/issues/7297) proposed the test directly: **if within-run CV ≫ across-run CV for a metric, that metric is cache-dominated.**

Applied to `onboardingImportWallet.total`, the numbers run the other way — 862ms between-run sd against 120ms within-run ([#45205 restore benchmark gate signal](https://github.com/MetaMask/metamask-extension/issues/45205)). By this test that metric is *not* cache-dominated, which is worth stating because it stops the cache finding from being used to explain every variance problem in the suite. The test needs running per metric, not assumed.

---

## Solution

Four steps. (1) is a prerequisite for judging (2); (3) and (4) come from [planning#7297](https://github.com/MetaMask/MetaMask-planning/issues/7297) and can proceed in parallel. All of them presuppose a live population exists to instrument.

1. **Record cache state per request, so hit and miss are separable after the fact.** A live path routed through mockttp as a pass-through lets a `beforeResponse` hook capture `cf-cache-status`, `age` and `cf-ray` per request and emit them alongside the timers. Without it, no claim about cache effects on CI runners can be checked — including the ones in this ticket.
2. **Keep the live population off the per-commit path.** Its purpose — confirming that the mocked delays still track reality, and catching accumulated small regressions — tolerates noise, because it averages over many runs. The per-commit path does not. [#45679 mock `main` and `release/*` too](https://github.com/MetaMask/metamask-extension/pull/45679) removed it; a cron that runs the live suite on a fixed schedule and alarms on a trend restores the purpose without putting CDN state in the way of a merge.
3. **Classify every endpoint the benchmark touches**, into `bypassable` (honors `Cache-Control: no-cache` or a `?ts=` buster), `CDN-fronted` (cached upstream of any header we control), or `app-server-cached` (in-memory, unreachable from outside). Produce the table in `docs/quality-gates/network-cache-inventory.md`; the `docs/quality-gates/` directory does not exist at `0350a018`. Where a test allows it, per-iteration cache-key salts such as randomized account addresses reach some of the third class.
4. **Tag each iteration `cachePhase: 'cold' | 'warm'`** — first iteration of a run cold, the rest warm — so cold-path regressions surface even when warm-path numbers look healthy. Gate `cold.*` variants where the distinction matters.

Not proposed: pinning the cache. A `cb=` buster forces misses and a warm-up pass forces hits, but neither number is the user-facing latency. That is a calibration call for [#45205 restore gate signal](https://github.com/MetaMask/metamask-extension/issues/45205), made with (1)'s data in hand.

---

## Acceptance Criteria

- [ ] `cf-cache-status`, `age` and `cf-ray` are captured per live request and land in the run's output
- [ ] A single run's results can be partitioned into cache-hit and cache-miss requests without re-running it
- [ ] The measured hit ratio on CI runners is reported, confirming or refuting the "mostly hitting cache" report with our own data
- [x] The live suite no longer runs on the per-commit path — [#45679 mock `main` and `release/*` too](https://github.com/MetaMask/metamask-extension/pull/45679), merged 2026-08-24
- [ ] A scheduled live run exists, its output feeds drift detection, and its failure notifies without blocking a merge
- [ ] Revalidation of the mocked delays against the live series is possible from that scheduled run
- [ ] `docs/quality-gates/network-cache-inventory.md` exists, with every benchmark-touched external endpoint classified
- [ ] Each `bypassable` endpoint has a cache-buster applied on the live path
- [ ] Each iteration emits a `cachePhase` tag
- [ ] Threshold registry has `cold.*` variants where cold-path matters (at minimum the page-load and swap-quote families)
- [ ] The within-run vs across-run CV comparison is run per gated metric and the result recorded in the inventory
- [ ] Any endpoint that is both unbypassable and cache-sensitive has a ticket filed against the owning service

---

## Labels

`team-extension-platform`, `area-testSuite`, `area-CI`

---

## Dependencies

**Supersedes:** [planning#7297 backend caching distorts real-network benchmark measurements](https://github.com/MetaMask/MetaMask-planning/issues/7297), closed 2026-08-12; its failure modes, workstreams and acceptance criteria are carried forward above.
**Blocked by:** a live population existing at all — [#45455 gate per-commit runs on mocked, measure live on a schedule](https://github.com/MetaMask/metamask-extension/pull/45455) (open) and [#45472 no stable real-world benchmark measure](https://github.com/MetaMask/metamask-extension/issues/45472). Nothing here is instrumentable while every run is mocked.
**Related:**

- [#45472 no stable real-world benchmark measure once live runs leave the per-commit path](https://github.com/MetaMask/metamask-extension/issues/45472) — the release-over-release series; this ticket is drift detection
- [#45204 stop gating the live-network benchmark population against mocked ceilings](https://github.com/MetaMask/metamask-extension/pull/45204), closed unmerged 2026-08-13 in favor of mocking every run, and [#45455 mocked per-commit, live on a schedule](https://github.com/MetaMask/metamask-extension/pull/45455) — the population split
- [#45205 restore benchmark gate signal](https://github.com/MetaMask/metamask-extension/issues/45205) — ceilings re-derived from CI data
- [#45431 benchmark runs are not observable](https://github.com/MetaMask/metamask-extension/issues/45431) — run and attempt identifiers, needed to join a cache-status record to its run
- [#45451 `main` benchmark baseline publishing stopped at GitHub's 100 MB file limit](https://github.com/MetaMask/metamask-extension/issues/45451) — baseline publishing, which a scheduled series also uses
- [planning#7280 fix mis-scoped onboarding password-transition timers](https://github.com/MetaMask/MetaMask-planning/issues/7280), [planning#7204 aggregated benchmark statistics average mean, stdDev and percentiles independently across runs](https://github.com/MetaMask/MetaMask-planning/issues/7204) — older harness findings this measurement bears on

Read test/e2e/benchmarks/utils/mock-config.ts, test/e2e/benchmarks/mocks/performance-mocks.ts, and .github/workflows/run-benchmarks.yml, noting that live instrumentation is blocked until a live population exists. Done means live requests emit cache headers and cachePhase data, scheduled runs report cache-hit/miss results, and the endpoint inventory is added at docs/quality-gates/network-cache-inventory.md.

Written by the indexing model from the issue text.