MetaMask / MetaMask/metamask-extension

[P2] Write incident postmortem and telemetry-quota incident runbook

Open
#43,413 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

INVALID-ISSUE-TEMPLATE ta-triaged team-extension-platform
Dominant language
TypeScript
Stars
13.2k
Forks
5.6k
Avg merge
2d 5h
Merged PRs (30d)
451

Description

**Parent Epic:** [#43410 - Sentry Quota Breach Incident — Extension Telemetry](https://github.com/MetaMask/metamask-extension/issues/43410)
**Labels:** `team-extension-platform`
**Priority:** P2

---

## 🎯 Goal

Publish a postmortem for the May 28 – Jun 2026 Sentry quota breach and distill it into an on-call runbook for future telemetry-quota incidents.

---

## 📝 Context

#43211 is labeled `RCA-needed`. The incident surfaced several hard-won, non-obvious facts that will be lost without a written record:

- The quota is measured in **performance units**, not raw spans; the Extension's budget is its allocated slice (30M/month).
- Telemetry from already-shipped releases can only be blocked server-side via a **release inbound filter**, which drops **all data types** for that release (errors, logs, …) — irreversible data loss is the price of stopping a runaway release.
- SDK-side `tracesSampler` fixes are forward-only; they cannot retroactively throttle installed builds.
- Sentry support initially gave incorrect guidance (inbound filters apply to errors only) that was later corrected — the decision tree should be written down so we don't re-litigate it under time pressure.

### Verified additions (2026-07-01, from the elevation/drain re-analysis)

- **Hard proof the release inbound filter drops spans + transactions (not just errors)** — Sentry outcomes ledger (90d): filtered/`release-version` = **56.2M spans + 22.0M transactions** + 9.5M errors. Put these numbers in the decision tree so the "errors-only" myth doesn't resurface a third time.
- **Scope per-release deliberately (all-or-nothing).** This incident: **13.32.0** was blocked (accepting total data loss — EOL'd), while **13.33.0** was *deliberately spared* the blanket filter to keep its errors/logs and left to adoption drain. Runbook rule: blanket-filter only releases you're willing to go fully dark on.
- **Apply at onset — watch for propagation lag.** 13.32.0 ran ~1 week at full span volume before dropping to 0; whether that was filter-application lag or pre-filter adoption is **unverified** (audit log needs org-admin scope). Open for the Sentry liaison: (a) all-or-nothing semantics, (b) filter-propagation lead time.
- **Outcome / timeline datapoint.** The acute assets-controller spike (`AssetsDataSourceTiming` 2216M→21M/wk across June, extrapolated) was ~99% drained by ~2026-07-01 via filter + forced-update adoption. SDK disable #43213 (MERGED cp-13.34.0) covers 13.34+; the `tracesSampler` (#43234 / PR #43228) is still OPEN/unmerged.

---

### Server-side control facts (verified Jun–Jul 2026)

- Release inbound filters apply to spans and transactions, all-or-nothing per release (support-confirmed, #43226).
- Measured filter application latency: ~3.8 days from filing (2026-06-04) to activation (mid-day 2026-06-08); once applied, the cutover lands within hours.
- Server-side rate limits apply to errors only (project limit 330 events/min ≈ 475K/day); transactions/spans are limitable only client-side (`tracesSampler`, `beforeSendSpan`).
- "Rate limited" spans in Stats & Usage indicate billing-cycle quota exhaustion; spend allocation reserves a minimum event volume and is not a cap.
- Sentry is not in the on-call escalation path.

## ✅ Acceptance Criteria

- [ ] Postmortem covering timeline (May 28 rate limiting → Jun 3 trial → mitigations), all three root causes (13.32.0 fan-out, 13.32.1 incomplete patch, 13.33.0 webpack filtering regression), and contributing factors (no quota alerts, undersized allocation).
- [ ] Runbook: how to detect quota exhaustion, the server-side vs SDK-side decision tree, who has Sentry admin access, and escalation paths (FireHydrant, Sentry support).
- [ ] `RCA-needed` resolved on #43211.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository files or tests are named; start with parent epic #43410, RCA issue #43211, and the linked incident facts in this issue. Done means publishing the accepted timeline, root causes, decision tree, access and escalation paths as a postmortem and runbook, then resolving RCA-needed on #43211.

Written by the indexing model from the issue text.

Assessment

Domain
documentation, observability-sre
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.