MetaMask / MetaMask/metamask-extension
[P1] Autonomous per-transaction-name span budget (circuit breaker) in `tracesSampler`
- Dominant language
- TypeScript
- Stars
- 13.2k
- Forks
- 5.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 451
Description
**Parent Epic:** [#43410 - Sentry Quota Breach: Extension Telemetry](https://github.com/MetaMask/metamask-extension/issues/43410)
**File:** `app/scripts/lib/setupSentry.js`
**Size:** M
---
## Problem
Two quota incidents have now been caused by the same shape of defect: a custom span emitted on a hot path with no sample gate, introduced accidentally.
- `AssetsDataSourceTiming` and `AssetsUpdatePipeline` (13.32.0, May, #43211).
- `AggregatedBalanceSelector` (13.38.0, July, #44447), which reached 91% of all transaction roots.
Both originate from the same upstream changeset ([MetaMask/core#8310](https://github.com/MetaMask/core/pull/8310)). Neither was added deliberately as telemetry. #43635 was a balance-correctness fix that incidentally forwarded a `trace` argument into a Redux selector.
The extension has no client-side bound on how much any single transaction name may contribute. A new span on a hot path can reach an arbitrary share of total volume, and nothing in the client stops it.
Every existing control requires knowing in advance which span is the problem:
- `tracesSampler` (#43234) assigns rates per transaction name, which requires naming the offender.
- The remote-flag controls (#7361, #7231, #7359, #7360) change rates without a release, which also requires naming the offender.
- The PR guardrail (#43414) depends on a reviewer recognising the pattern in review.
None of these bound an unknown future offender. In both incidents the offender was identified only after it had reached a large share of production volume, roughly a week later in each case.
---
## Solution
Add a per-transaction-name budget to `tracesSampler`, applied before the configured rate.
Sample the first N occurrences of a given transaction name per session at the normal rate, then decay the rate (for example, 1-in-M thereafter). Any transaction name is then bounded in its contribution regardless of how often the underlying code path executes, with no advance knowledge of which name will misbehave.
Properties:
- **Head-based.** The decision is made at trace start, so the span is never created and the client does no work for spans that would be discarded.
- **Autonomous.** Once deployed it requires no flag, no server, and no release to bound a new offender.
- **Diagnostic.** When a name exceeds its budget, tag the trip with the name and count, so the offender is identifiable from the retained sample rather than by manual investigation.
This bounds volume. It does not replace the remote-flag controls (#7361), which remain the only way to change behaviour on builds already installed. This is a floor, not a substitute.
---
Budget denominator: enforce per-name budgets against the legacy Discover "Transactions" (metrics-enhanced) dataset — per-name `count()` is billing-equivalent (metrics extracted before dynamic sampling; all sampling is client-side). Verify once against Stats & Usage accepted before first use. EAP/Explore counts are extrapolated (~60x) and must not be used for budget math.
## Sizing
`AggregatedBalanceSelector` fires per account group per Redux state change, so it concentrates in accounts with many groups. Approximately 12% of sampled users carry the transaction, yet it accounts for approximately 60% of all span volume. A per-name budget bounds exactly this shape: heavy emitters are capped, ordinary usage is unaffected.
---
## Acceptance Criteria
- [ ] `tracesSampler` applies a per-transaction-name budget before the configured rate.
- [ ] No single transaction name can exceed a defined share of a client's span emission per session.
- [ ] Budget trips are tagged so the offending name is identifiable in the retained sample.
- [ ] The `AggregatedBalanceSelector` emission pattern, replayed against the budget, is bounded.
---
## Labels
`team-extension-platform`, `area-sentry`
---
## Dependencies
**Depends on:** #43234 (the `tracesSampler` mechanism, PR #43228 still open).
**Complements:** #7361 (remote-flag sampling control, the only lever that reaches installed builds), #43414 (PR-review guardrail).
**Detection gap.** #43411 covers alerting on rate-limited and discarded volume, but not per-transaction-name volume anomaly. Neither #43211 nor #44447 was surfaced by volume alerting, and both ran for roughly a week. An alert on a new transaction name appearing with material volume share would have caught both within hours. This should be added to #43411's scope.
Contributor guide
Research direction
Start in app/scripts/lib/setupSentry.js and read the tracesSampler mechanism from dependency #43234 before deciding how the per-transaction-name budget fits the existing configured rate. Define and verify the session budget, decay behavior, and diagnostic tags against the stated dataset guidance. Done means the acceptance criteria pass, including replaying the AggregatedBalanceSelector pattern and proving no name exceeds the defined share.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100