MetaMask / MetaMask/metamask-extension

Multichain account creation fan-out returned to 6x its post-fix level

Open
#45,574 1 comment 0 reactions 0 assignees View on GitHub
INVALID-ISSUE-TEMPLATE Sev2-normal ta-needs-engineer-escalation ta-triaged team-accounts-framework
Dominant language
TypeScript
Stars
13.2k
Forks
5.6k
Avg merge
2d 5h
Merged PRs (30d)
451

Description

**Parent Epic:** [#43410 - Sentry Quota Breach Incident — Extension Telemetry](https://github.com/MetaMask/metamask-extension/issues/43410)
**Follows:** [#43633 - Investigate and cap multichain account creation/syncing span fan-out](https://github.com/MetaMask/metamask-extension/issues/43633) (closed 2026-07-15)
**Source:** `@metamask/multichain-account-service` — account providers + `MultichainAccountWallet` alignment loop
**Size:** M | **Priority:** P1

---

## Problem

`Provider Create Accounts (v2 - batched)` fan-out is back above its pre-fix level, and its share of non-spike transaction volume has roughly five-tripled since June.

Measured 2026-08-17, project `metamask` (273505), production, matched 4-day windows. Shares are computed on a **spike-excluded denominator** — `AggregatedBalanceSelector`, `AssetsDataSourceTiming` and `AssetsUpdatePipeline` removed — because the raw denominator was up to 89% spike volume in July and masks this family entirely.

| Window | Transactions per trace | Share of non-spike volume |
|---|---|---|
| 19–23 May | 1.83 | 2.75% |
| 16–20 Jun | **1.50** | 3.07% |
| 14–18 Jul | 2.39 | **11.77%** |
| 4–8 Aug | **9.51** | **15.74%** |
| 13–17 Aug | 4.38 | 9.61% |

Two independent signals move together:

- **Fan-out** rose from **1.50 to 9.51 transactions per trace** — 6.3× the June level. This ratio is unaffected by sampling, since the retention decision is taken per trace and is constant within one, so it is directly comparable across windows.
- **Share of non-spike volume** rose from 3.07% to a peak of 15.74%, making this the largest single transaction family outside the service-worker pageload.

The inflection sits between mid-June and mid-July, around the period [#43633 (the original fan-out cap)](https://github.com/MetaMask/metamask-extension/issues/43633) was closed — not months later.

## Why this was not visible sooner

On the raw denominator the family appears to *fall* through this period — 2.68% in May to 1.24% in mid-July — because the assets-controller spike inflated total volume roughly 9.5× in that window. Any share computed against raw totals during a spike understates every other family. The spike-excluded basis above is the one to use for this class of question.

---

## What to check

- Whether the cap introduced for [#43633 (the original fan-out cap)](https://github.com/MetaMask/metamask-extension/issues/43633) is still in effect on current releases, or was reverted, bypassed by a new call path, or scoped to a code path that has since moved.
- Whether the alignment loop re-entered a per-account rather than per-batch pattern — a fan-out of 9.5 per trace suggests one root per account rather than per batch.
- ~~Whether this is release-correlated~~ — answered below.

---

## Root cause found — Sentry SDK v10 native trace propagation (2026-08-17)

`@metamask/multichain-account-service` never changed: `yarn.lock` resolves the exact same `11.1.0` across `13.40.0`, `13.41.0`, and `13.42.0`. The fix is not in account logic at all — it's in the extension's own Sentry integration.

[PR #42867 (Sentry SDK v10 upgrade)](https://github.com/MetaMask/metamask-extension/pull/42867) (commit `7bfc16c`, merged 2026-07-17, shipped in the `13.42.0` build) upgraded `@sentry/browser` 8.33.1 → 10.38.0 and, in `app/scripts/lib/sentry-trace-propagation.ts`, replaced this extension's own manual `traceparent` construction (marked `// TODO: Remove once sentry SDK is upgraded to v10` throughout the removed code) with the SDK's native `propagateTraceparent`. The manual implementation was the mechanism attaching outbound backend calls to a trace; the native replacement does the same job differently, and that difference is what changed fan-out.

This generalizes — it is not specific to `Provider Create Accounts (v2 - batched)`. The same `13.41.0` → `13.42.0` step-change appears in every backend-calling multichain-account family checked, aggregate window 4–8 Aug:

| Family | `13.41.0` fanout | `13.42.0` fanout |
|---|---|---|
| `Provider Create Accounts (v2 - batched)` | 12.75 | 4.19 |
| `Wallet Alignment` | 3.52 | 1.41 |
| `Create Solana Account Batch` | 3.51 | 1.36 |
| `Wallet Create Multichain Account Groups` | 3.51 | 1.27 |

`13.42.0` and `13.43.0` both hold in the same ~1.2–1.5 band (Provider v2 aside, which runs structurally higher given real per-account fan-out) in the 13–18 Aug window too — stable, not still declining. `13.41.0`'s own fan-out is unchanged whenever it appears (2.0–2.35 on small residual samples now) — the fix does not retroactively help stragglers still on that release; their share of traffic is just shrinking as they update, same as noted below.

This is a genuinely shipped fix, not a hidden regression working itself out. The release breakdown below predates this specific finding and is kept as-is — it independently establishes the same release-mix mechanism from the data side, before the code-side cause was identified.

## Release breakdown — the decline is attrition, not a fix (2026-08-17)

Broke fan-out down by `release` for the same two windows. This changes the read on the whole ticket: the aggregate recovery from 9.51 to 3.78–4.38 is **not** a fix taking effect. It's one bad release's share of traffic shrinking while a normal release's share grows.

| Release | Peak window (4–8 Aug) | Now (13–18 Aug) |
|---|---|---|
| `13.41.0` | fanout **12.75** (47,606 tx, 79% of this transaction's volume) | fanout **6.78** (902 tx, ~4% of volume) |
| `13.42.0` | fanout 4.19 (11,844 tx) | fanout 4.10 (16,244 tx) |
| `13.43.0` | *(not yet released)* | fanout 3.69 |

`13.42.0`'s own fan-out is flat — 4.19 during the peak, 4.10 now — while it sat directly alongside `13.41.0` both times. It was never elevated. `13.41.0`'s fan-out is still 3× normal **right now**, for whoever hasn't updated off it; what changed is that its share of traffic collapsed from 79% to ~4% as users moved to `13.42.0`/`13.43.0` in the ordinary release cadence.

This is superseded by the root cause section above: `13.41.0` shipped with the old, TODO-marked manual `traceparent` code, and `13.42.0` shipped [PR #42867 (Sentry SDK v10 upgrade)](https://github.com/MetaMask/metamask-extension/pull/42867)'s native replacement. The two facts aren't in tension — the fix landed in a specific release, and reaching the remaining users on `13.41.0` still happens by ordinary attrition, since there is no version-targeted kill-switch for this (the `transactionSampleRates` feature-flag mechanism doesn't reach general availability until `13.44.0`).

---

## Status — likely resolved, closing candidate

Root cause identified, fix already shipped and verified stable across two windows and four families. What's left is confirming there's nothing further to do, not more investigation:

- [x] Root cause identified: [PR #42867 (Sentry SDK v10 upgrade)](https://github.com/MetaMask/metamask-extension/pull/42867) — old manual `traceparent` construction, superseded by Sentry SDK v10 native `propagateTraceparent`
- [x] Confirmed the fix is real, not aggregate self-correction: release-pure fan-out on `13.42.0`/`13.43.0` is stable at ~1.2–1.5 (Provider v2 slightly higher, structurally, given real per-account fan-out) across both the peak window and now
- [x] Confirmed it generalizes beyond this one transaction — same step-change on `Wallet Alignment`, `Create Solana Account Batch`, `Wallet Create Multichain Account Groups`
- [ ] Confirm `13.41.0` adoption is genuinely trending toward zero rather than plateauing on a stuck cohort (enterprise pin, disabled auto-update, etc.) — if it plateaus non-trivially, that's a separate, smaller follow-up, not a reason to keep this open at P1
- [ ] A regression check that would catch a *new* propagation defect recurring, scoped per-release — the previous fix ([#43633](https://github.com/MetaMask/metamask-extension/issues/43633)) closed without one, and this regression was found by unrelated volume analysis a month later
- [ ] Candidate to close once the above is confirmed, cross-referencing this finding from [#43410's postmortem](https://github.com/MetaMask/metamask-extension/issues/43413) so the mechanism (a shared tracing-layer change, not app-logic) isn't lost

---

## Labels

`team-accounts-framework`, `area-sentry`, `area-multichain`

Contributor guide

Open the contributing guide

Research direction

Read app/scripts/lib/sentry-trace-propagation.ts and PR #42867 to understand the tracing change from manual traceparent construction to Sentry SDK propagation. Then verify whether 13.41.0 adoption is approaching zero and determine a per-release regression check for future propagation changes. Done means confirming the remaining checklist or documenting a separate follow-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.