MetaMask / MetaMask/metamask-extension
Re-architect assets-controller `AssetsDataSourceTiming` / `AssetsUpdatePipeline` instrumentation to be quota-safe before re-enabling traces
- Dominant language
- TypeScript
- Stars
- 13.2k
- Forks
- 5.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 451
Description
### What is this about?
The assets-team-owned, root-cause half of the span-quota incident (#43211). The two custom transactions the `AssetsController` emits — `AssetsDataSourceTiming` and `AssetsUpdatePipeline` — fan out across every data source × chain × account on every poll cycle (≈367M + ≈81M spans/24h on 13.32.0; still ≈24M + ≈4.2M on patched releases), which drained the `metamask` project's Sentry perf-unit slice.
Everything shipped so far only contains the symptom at the telemetry layer:
- #43213 removed `trace: traceAsControllerCallback` from the controller init — a blunt source disable; all timing/pipeline observability is lost.
- #43234 / PR #43228 pins both names to `0` in the client `tracesSampler` — suppresses them by name, also losing the signal.
- #43232 adds a runtime lever to re-budget those rates without a release — still only a throttle.
None of these restore the observability the spans were meant to provide. This ticket is the assets team reshaping the instrumentation so it answers the question it was added for without re-breaching the budget — the prerequisite for ever turning `trace` back on.
### Why
Diagnostic, not aspirational. The measurements are useful (data-source latency, update-pipeline timing) but were emitted as one Sentry transaction per operation in a polling hot path — `balanceInterval` / `stakedBalanceDataSourceConfig` poll every 30s, `detectionInterval` / price `pollInterval` every 180s — then multiplied by chains and accounts. That fan-out is what breached quota, not raw event volume. Capping or disabling by name (the platform mitigations) trades the breach for a blind spot. The durable fix has to live here because only the assets team can decide which measurements matter and at what granularity, then reshape the emission to match.
### Scope / approach (sketch)
The fix is whatever makes the measurement quota-safe. Candidate directions, not all required:
- Collapse the fan-out: one span per logical pipeline run, not one per data-source × chain × account × poll. Push detail into attributes, not into separate transactions, to cut cardinality.
- Move always-on polling latency off Sentry transactions onto the metrics / Segment path (the Tier-3 option, mirroring network-controller degraded-endpoint monitoring). Sentry tracing is the wrong tool for high-frequency steady-state timing.
- If it stays on Sentry, ship at a deliberate sub-sample rate coordinated with platform's `SENTRY_SAMPLE_RATE_OVERRIDES` (#43234) so any re-enable rides a budgeted rate, never 100%.
- Ship with a kill switch and a sub-sample gate on day one — the standing lesson from this incident.
### Acceptance Criteria
- [ ] A stated, measured perf-unit budget for the assets instrumentation, validated against the `metamask` allocated slice
- [ ] Redesigned emission proven to stay within that budget under realistic multi-chain / multi-account polling (no per-poll × per-source fan-out)
- [ ] The original observability question the spans answered is still answerable after the redesign — or explicitly dropped, with rationale
- [ ] Re-enable path defined: `trace: traceAsControllerCallback` (or its replacement) reintroduced only behind a sub-sample gate + kill switch, at the coordinated rate
- [ ] The platform name-pin (#43234) and runtime lever (#43232) revert to guardrails — neither is the steady-state control for these spans
### References
- #43211 — origin incident (sev-1 release blocker)
- #43213 — disable assets controller traces at source (the blunt containment this replaces)
- #43234 — client-side build-time `tracesSampler` name-pin (platform guardrail; derives from)
- #43232 — runtime re-budgeting via remote flags (platform guardrail; derives from)
- #43226 — server-side inbound filter for shipped releases (related)
- `app/scripts/messenger-client-init/assets/assets-controller-init.ts` (`trace` wiring) · `@metamask/assets-controller` (span emission) · `shared/lib/trace.ts` (`traceAsControllerCallback`)
Contributor guide
Research direction
Start with app/scripts/messenger-client-init/assets/assets-controller-init.ts and shared/lib/trace.ts, then trace the AssetsDataSourceTiming and AssetsUpdatePipeline emission in @metamask/assets-controller. Define the measurement budget and chosen granularity before implementation; done means realistic multi-chain and multi-account polling stays within budget, the observability question remains answerable, and re-enabling trace is protected by a sub-sample gate and kill switch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- observability
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100