[Epic] Sentry Quota Breach — Extension Telemetry (May 2026 – ongoing)

**Labels:** `epic`, `team-extension-platform`, `Sev1`

---

## 🎯 Goal

Resolve the **recurring** Sentry telemetry quota breaches (May 2026 – ongoing) and future-proof the Extension via sampling controls, runtime re-budgeting, quota alerting, and instrumentation guardrails.

> ### ⚠️ Scope broadened 2026-07-14 — this is not a single May–Jun incident
>
> Volume has **never returned to baseline.** It has been sustained by a **rotating cast of offenders** — each fix followed by a new ungated hot-path span shipping:
>
> **assets-controller** (13.32.0, May–Jun — drained) → **multichain account fan-out** (rising through June — fixed, shipped 13.38.0) → **`AggregatedBalanceSelector`** (13.38.0, July — now **91% of all transaction volume**, #44447).
>
> Accepted transactions are still **~36M/week against a ~3.2M/week pre-spike baseline (~11x)** and *rising* with 13.38+ adoption. The multichain fix should have cut ~37% of volume; `AggregatedBalanceSelector` backfilled it entirely **in the same release**.
>
> **The systemic gap: nothing prevents a new ungated hot-path span from shipping** — the guardrails meant to stop exactly this (#43370, #43414) were bypassed.

---

## 📝 Incident Summary

- The production Extension Sentry project (`metamask`) exhausted its monthly **spend-allocation slice** — denominated in **performance units, not raw spans** — triggering rate-limiting and dropped incoming telemetry from **May 28 – Jun 3 2026** ([Stats & Usage](https://metamask.sentry.io/settings/stats/?clientDiscard=false&dataCategory=transactions&project=273505&statsPeriod=14d&transform=periodic)). A Sentry trial temporarily restored collection.
- The budget to engineer against is **`metamask`'s allocated perf-unit slice** — reduce consumption to fit it; raising the allocation is break-glass, not the plan. Raw span counts below only *rank* offenders; absolute "do we fit?" math needs perf-units from Stats & Usage.
- Acute peak (early Jun): **578M spans/24h from 13.32.0** and **162M from 13.33.0** — the assets-controller and webpack `http.client` regressions.
- As the acute offenders are filtered/drained, residual consumption on the **healthy current release** is dominated by two **structural** sources: **`/service-worker.js` cold-starts (~45% of span volume)** and **multichain account discovery/alignment (~20%, rising to ~44% as the acute fixes drain)** — see Root Causes and the Mitigation Approach table.
- The only lever against already-installed builds is a server-side release inbound filter, which drops **every data type** for a release (errors, logs, spans) — irreversible data loss. Recurrence risks release blockers and further loss.

---

## 🔬 Root Causes

Four independent acute causes — three in May–Jun, a **fourth in July**:

1. **13.32.0 — assets-controller instrumentation with high fan-out.** `AssetsDataSourceTiming` / `AssetsUpdatePipeline` custom spans fan out per trace (~578M spans/24h ≈ all of 13.32.0). #43211.
2. **13.32.1 — incomplete assets-controller patch.** Reduced fan-out but did not curtail the excess.
3. **13.33.0 — span-filtering regression from the webpack migration.** ~20M/day local-file fetch spans (`http.client`) filtered in #41526 are traced again by the webpack build. (#39891 background-RPC tracing was suspected as a compounding factor; ruled out in #43629 — its wrapper spans are ~0.49% of post-fix volume.)
4. **13.38.0 (July 2026) — `AggregatedBalanceSelector`: an ungated trace on a hot-path Redux selector.** `@metamask/assets-controller@3.2.0` (the **same [MetaMask/core#8310](https://github.com/MetaMask/core/pull/8310)** that shipped cause (1)) added an optional `trace` param to `getAggregatedBalanceForAccount`; the extension passes it **ungated** at `ui/selectors/assets.balance-utils.ts:56`. A Redux selector recomputes on every relevant state change / re-render, per account group → each computation emits a forced transaction root (`span.op: custom`). **91% of all transaction roots** (~1.4–1.5B extrapolated/7d). **Same core PR and anti-pattern as (1):** #8310 shipped both the assets-controller spans *and* this selector param; #43213 disabled only the controller-callback path, leaving the selector-param path live to detonate months later (incidentally armed by balance-correctness PR #43635). The #43370 guardrail was bypassed. **No server-side lever this time:** no transaction-name inbound filter exists, and the release filter is all-or-nothing on the current train. Code-only fix (**#44449, cp-13.40.0**). **#44447.**

Two **structural** sources dominate the *post-fix* healthy release and outlast the acute offenders:

- **`/service-worker.js` MV3 cold-start volume** — ~45% of healthy-release span volume. Sentry breakdown (`count(span.duration)`, prod, project 273505, 30d, `transaction:/service-worker.js` — [explore](https://metamask.sentry.io/explore/traces/?aggregateField=%7B%22groupBy%22%3A%22transaction%22%7D&aggregateField=%7B%22yAxes%22%3A%5B%22count%28span.duration%29%22%5D%7D&environment=production&mode=aggregate&project=273505&statsPeriod=30d&query=transaction%3A%2Fservice-worker.js)): **2.18B spans** that decompose into ~**55% operational `http.client`** (usage-driven RPC — count invariant to the SW lifecycle), ~**30% boot `http.client`** re-run on each of 146M cold-starts (geolocation, supportedNetworks, static-config, infura init — the MV3 multiplier), and ~**13%** per-cold-start `pageload` + `sentry-tracing-init` overhead. (The headline 87%/1.89B `http.client` is the **total** fetch volume, boot **and** operational; only the ~30% boot subset is MV3-reducible. `http.client` is `browserTracingIntegration` fetch auto-instrumentation, not #39891 — ruled out, #43629.) **This is a model mismatch, not excess spans** — a document integration applied to an ephemeral task runner — and it is fixed by **two reworks, not by trimming**: (1) **app-architecture** — stop re-bootstrapping startup per cold-start (`MetaMask/MetaMask-planning#7366`, the ~30% boot-fetch volume lever, an app-behavior change), and (2) **tracing-model** — bound the SW root and root operations as tasks (`MetaMask/MetaMask-planning#7354` / `#7355`, fixes the 445K–535K mega-trace *shape*, preserves the ~55% operational coverage + its volume). The incident-owned trims — drop `sentry-tracing-init` marks (#43960, 6.7%) and the `http.client` noise subset segment.io + static-config (#43235) — are **stopgap relief, not the fix**.
- **Multichain account discovery/alignment fan-out** — ~20% rising to ~44% as the acute fixes drain in; an always-on baseline back to 13.30, from per-index re-tracing of existing accounts (snap providers `batched: false`) (#43633). **✅ LARGELY FIXED — shipped in 13.38.0 (snap providers switched to `batched: true`).** The unbatched per-index fan-out is eliminated: `Provider Create Account (v1)` **107.6M → 0** and `Create Bitcoin Account` **51.3M → 0** on 13.38+, replaced by batched equivalents (`Provider Create Accounts (v2 - batched)` 12.3M → 19.6M; `Create Bitcoin Accounts Batch` new at 6.7M). Family total **~222.5M → ~52.8M (~76% reduction — reduced, *not* zeroed;** the batched v2 + alignment/syncing/contact-sync remain). Width confirmed: `Wallet Alignment` 70.9M triggers vs `Provider Create Account (v1)` 485.7M = **~6.85 per alignment**, i.e. per-index re-tracing of already-existing accounts. **The gain was immediately consumed by `AggregatedBalanceSelector` in the same release (#44447), so net volume did not move.**

---

## 📋 Tickets

### Phase 1 — Mitigation (stop the bleed)

| # | Ticket | Priority |
|---|--------|----------|
| 1 | #43211 Assets-controller instrumentation exceeding quota — disable at source (PR #43213, cp-13.34.0) | P0 |
| 2 | #43226 Server-side release inbound filter for shipped releases + drain | P0 |
| 3 | #43235 `/service-worker.js` `http.client` span volume — drop local fetches + segment.io + static-config (PR #43236, cp #43241); startup-burst sub-sample | P1 |
| 4 | **#44447 `AggregatedBalanceSelector` span fan-out — 91% of all transaction volume; ungated `trace` on a hot-path balance selector (13.38.0+). Code-only fix (no inbound-filter lever); cherry-pick.** | **P0** |

### Phase 2 — Follow-up fixes

| # | Ticket | Priority |
|---|--------|----------|
| 4 | #43234 Config-driven Sentry `tracesSampler` — cap high-volume custom transactions by name (PR #43228); the sampler mechanism | P1 |
| 5 | #43654 Drop the `sentry-tracing-init` mark + other low-diagnostic-value marks via `beforeSendTransaction` (PR #43960) — ~30M/7d, all transactions | P2 |
| 6 | #43633 Cap multichain account creation/syncing span fan-out — #1 remaining source (~20% → ~44%), always-on baseline back to 13.30 | P0 |
| 7 | #43370 Re-architect assets-controller instrumentation to be quota-safe before re-enabling traces | P1 |

### Phase 3 — Prevention & early response

| # | Ticket | Priority |
|---|--------|----------|
| 8 | #43411 Sentry quota alerting — rate-limited/discarded telemetry + spend-allocation exhaustion | P1 |
| 9 | #43631 Freeze OTLP backend-span ingest onboarding until quota under control | P1 |
| 10 | #43412 Configure spike protection & rate limits; RCA why spend allocation was exceeded | P2 |
| 11 | #43413 Incident RCA postmortem + telemetry-quota runbook | P2 |
| 12 | #43414 PR-review guardrail for new span instrumentation (fan-out, sample gates, kill-switches) | P3 |
| 13 | **#44452 Autonomous per-transaction-name span budget (circuit breaker) in `tracesSampler` — bounds an offender *before* it is named; the durable answer to the rotating-offender pattern** | P1 |
| 14 | **#44477 Restore/re-optimize global `tracesSampleRate` once dynamic sampling controls ship — the strategic payoff of the remote flag** | P2 |

### Investigation

| # | Ticket | Priority |
|---|--------|----------|
| 15 | #43629 Rule out PR #39891 (background-RPC distributed tracing) as a contributing factor | P3 |
| 16 | #44450 Balance selector recomputes 4x with flat inputs (2026-07-13, 14h) — likely a balance-state regression behind the telemetry defect; assets team | P2 |

> Remote-flag sampling and `/service-worker.js` pageload re-rooting are **owned by the tracing program** (`MetaMask/MetaMask-planning#7231` + `MetaMask/MetaMask-planning#7359` + `MetaMask/MetaMask-planning#7360`, `MetaMask/MetaMask-planning#7354`/`MetaMask/MetaMask-planning#7355`) — see Program Coordination. Incident duplicates `#43232`, `#43655` are closed/superseded into them; `#43654` is **rescoped** to the low-value-mark drop (active, Phase 2 — complements the program's SW re-rooting).

---

## 🌳 Dependency Tree

```
#43211 (disable instrumentation) ──┬─→ #43234 tracesSampler ─┬─→ #43633 multichain account fan-out cap (tail-preserving)
#43226 (inbound filter + drain) ───┘ └─→ #43370 assets-controller re-architecture
#43235 (SW http.client startup-burst) ─→ #43412 spike-protection / rate-limit config
#43629 (#39891 ruled out) ─→ #43413 RCA postmortem
#43411 quota alerting — independent
#43631 OTLP ingest freeze ← gated on #43411 + #43414
#43413 RCA postmortem + runbook ← consumes all of the above
#43414 PR-review guardrail ← informed by RCA

Structural / durable work owned by the tracing program (see Program Coordination):
SW pageload re-rooting ──── MetaMask/MetaMask-planning#7354 / MetaMask/MetaMask-planning#7355 (supersedes #43654's root-sampling; #43654 rescoped to mark-drop, complement)
remote-flag sampling ────── MetaMask/MetaMask-planning#7231 + MetaMask/MetaMask-planning#7359 + MetaMask/MetaMask-planning#7360 (supersedes #43655 → MetaMask/MetaMask-planning#7359, #43232 → MetaMask/MetaMask-planning#7360)
http.client fetch filter ── MetaMask/MetaMask-planning#7168 (closed) ← extended by #43235
```

---

## 🧭 Mitigation Approach

### Current containment — July offender (2026-07-15)

`AggregatedBalanceSelector` (#44447) is code-only: no server-side lever reaches a rogue transaction on the current release train (no transaction-name inbound filter exists; the release filter is all-or-nothing per release; the DSN key rate-limit is errors-only). Three levers, by reach:

1. **Code fix, forward.** #44449 removes the span, **cherry-picked to 13.40.0** (`cp-13.40.0` — target set in the PR title, not a label). Reaches only builds that ship it, so the effect arrives with 13.40.0 adoption. **13.40.0 is the clean landing build.**
2. **Global rate cut, forward.** `tracesSampleRate` reduced **0.75% → 0.5%** as interim headroom. Client-side, so it also reaches only 13.40.0+, not the installed 13.38/13.39. Protects the fixed build's budget, not the current spike. Restoration and re-optimization tracked in **#44477**.
3. **Release inbound filter, installed builds.** The only lever for already-installed 13.38/13.39. **Plan: filter 13.38/13.39 on 2026-07-24** when the Sentry trial expires, unless a further extension is granted (third extension, unlikely). All-or-nothing, so it drops all telemetry (errors, logs, spans) on those builds. Safe to apply **only because 13.40.0 (cp'd, clean) is the build users land on** — the sequencing lesson from the 13.32→13.33 drain: filter a release only once its successor is clean.

**Timeline (resolved).** 13.40.0 ships in **Thursday 2026-07-16's weekly release**, and patches release within the same week, so the fix has a fast path regardless of the cut. That gives roughly **8 days of adoption before the 2026-07-24 (Friday) trial expiry**. Chrome auto-update reached majority in ~5 days for 13.38.2, so 13.40.0 (and 13.41.0, the clean 07-23 weekly release) will be the dominant builds well before the filter. The clean-successor requirement for filtering 13.38/13.39 is therefore met with margin — this is no longer the gating risk it appeared to be when the cp target was mistaken for absent.

### May offender — per-release levers

Per-release span volume (project 273505) frames the levers:

| release | spans / 7d | role |
|---|---|---|
| 13.33.0 | 1.11B | shipped offender, EOL via forced update — inbound-filter candidate |
| 13.34.1 | 692M (~63% `http.client`) | healthy current release — cannot inbound-filter (only good build) |
| 13.32.1 | 52M | EOL — optional filter |
| 13.34.0 | 23M | superseded — optional filter |
| 13.35.0 | 11M | forward fixes landed — ~100× lower |

**Key constraint:** release inbound-filtering is **necessary but not sufficient** — the healthy current release is the 2nd-largest emitter and cannot be blocked without losing all observability on the only good build. Server-side proportional levers + break-glass cover the gap while forward fixes drain.

Planned sequence:

1. **DSN-level rate-limit + spike protection** (#43412) — proportional shed across all releases; preserves healthy-release samples, unlike all-or-nothing filters.
2. **Release inbound filter for the shipped offenders** (#43226) — 13.33.0 (+ optionally 13.32.1 / 13.34.0); **not** the healthy 13.34.1.
3. **Break-glass: trial extension or temporary spend-allocation bump** while forward fixes drain.
4. **Freeze OTLP backend-span onboarding** (#43631) — no new ingest source into an over-quota project; gate behind quota-under-control + #43411 + #43414.
5. **Forward fixes drain via forced-update to 13.35.x** — the #43234 sampler (#43633 multichain), #43235 / #43236 (`http.client`), and the program's SW pageload re-rooting (`MetaMask/MetaMask-planning#7354`).
6. **Dynamic sampling** (if Business/Enterprise) — relay-level `http.client` sampling; confirm plan support with Sentry.

---

## ✅ Success Criteria

- [ ] `metamask` consumption back under its allocated perf-unit slice
- [ ] DSN rate-limit + spike protection configured (#43412)
- [ ] Release inbound filter covers the shipped offenders; healthy 13.34.1 untouched (#43226)
- [ ] Break-glass decision made (trial extension vs allocation bump)
- [ ] OTLP backend-span onboarding gated until quota under control (#43631)
- [ ] `/service-worker.js` (program: `MetaMask/MetaMask-planning#7354`) and multichain-account (#43633) span volume reduced
- [ ] #39891 ruled out as a contributing factor (#43629)

---

## 🔗 Program Coordination

The durable, structural Sentry-tracing work surfaced by this incident is owned by the existing tracing program in `MetaMask/MetaMask-planning` (epics [MetaMask/MetaMask-planning#6759 L3: Sentry Attribution](https://github.com/MetaMask/MetaMask-planning/issues/6759), [MetaMask/MetaMask-planning#7238 OTel Trace Context Propagation](https://github.com/MetaMask/MetaMask-planning/issues/7238)). This incident epic owns the **acute** response and references the program for the structural fixes rather than duplicating them:

- **Remote-flag sampling** — `MetaMask/MetaMask-planning#7231` (flag mechanism + schema), `MetaMask/MetaMask-planning#7359` (release-level `tracesSampleRate` ceiling / emergency throttle), `MetaMask/MetaMask-planning#7360` (per-transaction-name overrides). Closed incident dups fold in: `#43655` → MetaMask/MetaMask-planning#7359, `#43232` → MetaMask/MetaMask-planning#7360.
- **`/service-worker.js` model rework (two streams)** — the SW volume is a model mismatch (a document integration on an ephemeral worker), fixed by **two reworks, not trimming**:
- *Tracing-model* — `MetaMask/MetaMask-planning#7354` / `MetaMask/MetaMask-planning#7355` (**bound** the SW root; per-operation + bounded-boot tracing; boot `MetaMask/MetaMask-planning#7341`, correlation `MetaMask/MetaMask-planning#7340`). Fixes trace *shape*; preserves coverage + volume.
- *App-architecture* — `MetaMask/MetaMask-planning#7366` (persist background state across cold-starts; stop re-bootstrapping startup per wake). The **87% volume lever**; an app-behavior change (per-fetch freshness/compliance review).
- Incident trims `#43654` → PR #43960 (drop marks) and `#43235` (`http.client` noise) are **stopgap complements**, not the fix.
- **`http.client` fetch-span filtering** — `MetaMask/MetaMask-planning#7168` (closed) — extended by `#43235`.
- **Sample-rate policy / SDK** — `MetaMask/MetaMask-planning#7249` (extension + mobile policy), `MetaMask/MetaMask-planning#6858` (SDK v8 → v10 — done via #42867, merged 2026-07-17).

---

## 📚 References

- Main thread: https://consensys.slack.com/archives/C0532K3LMA5/p1780418457240929
- Assets-team thread: https://consensys.slack.com/archives/C0AKX44EGH4/p1780577987968469
- 13.34.0-rc #43228 de-block: https://consensys.slack.com/archives/C0B5U5UM119/p1780592076868479
- 13.34.0-rc #43236 cp #43241: https://consensys.slack.com/archives/C0B5U5UM119/p1780604660775379
- Incident opening thread (FireHydrant): https://consensys.slack.com/archives/CTQAGKY5V/p1780657942902159?thread_ts=1780569185.918889&cid=CTQAGKY5V
- Sentry alerts for project (none configured for this scenario): https://sentry.io/organizations/metamask/monitors/alerts/?project=273505

Start with the dependency tree and select a specific child ticket rather than treating this epic as one change. For the current selector incident, read ui/selectors/assets.balance-utils.ts around line 56 and trace the Sentry transaction path; related work also references /service-worker.js and tracesSampler. Done means the selected offender is addressed, telemetry fits the allocated quota, and the relevant mitigation or guardrail is verified.

Written by the indexing model from the issue text.