MetaMask / MetaMask/metamask-extension
Sentry v10 migration: verify equivalent telemetry coverage (envelope capture-and-diff harness)
- Dominant language
- TypeScript
- Stars
- 13.2k
- Forks
- 5.6k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 451
Description
## Goal
Verify the Sentry v8→v10 upgrade (#42867) produces **equivalent telemetry coverage** — the same errors, transactions, tags, scrubbing, sampling, and no volume increase — rather than trusting a green test suite. "Equivalent" = v8 and v10 produce the same set of error events (same grouping, same sent/masked/removed state fields), the same named transactions with equivalent span trees, the same tag/context coverage, exactly one correct `traceparent` on backend calls, and no increase in envelope/perf-unit volume.
## Three-layer verification
**1. Snapshot-equivalence (already exists — use it as a gate).** Run `test/e2e/tests/metrics/*` on v10 against the committed v8 snapshots (`state-snapshots/errors-*`). Any required snapshot change is a behavioral delta to triage as **benign** (timing race, e.g. `pendingShieldCohort`/`srpSessionData`) vs **regression**. Do not blanket `--update-snapshot`; diff each.
**2. Envelope capture-and-diff harness (this ticket — the missing piece).** Snapshots only cover what's asserted. Build a harness that records the FULL set of Sentry envelopes for a fixed flow on v8 and v10 and diffs them.
Design (build on `getEventPayloads` / `sentryRegEx` in `test/e2e/helpers.js`):
- **Capture:** a mockttp handler on `sentryRegEx` with `.thenCallback(() => ({ statusCode: 200, json: {} }))` that records every POST body (don't filter by `withBodyIncluding`). Sentry posts are **newline-delimited envelopes**, not Segment `.batch` JSON — so add a dedicated parser.
- **Parse:** split the raw body on `\n` → line 0 is the envelope header (JSON), then repeating pairs of [item-header JSON, payload JSON]. Group into `{ type: item-header.type, payload }[]` (types: `event`, `transaction`, `session`, `client_report`, …).
- **Normalize:** strip volatile fields before compare — `event_id`, `timestamp`, `start_timestamp`, `sent_at`, `trace_id`, `span_id`, `parent_span_id`, `sdkProcessingMetadata`, durations, and any monotonic IDs. Sort items by `(type, transaction|exception.type|message)`.
- **Diff:** structural diff of normalized v8 baseline vs v10 current → report added/removed/changed envelope items, per-event tag/field deltas, and **envelope counts by type** (the quota axis).
- **Flow:** a fixed deterministic scenario — onboard → unlock → trigger a known error (developer options) → pageload → one backend `*.api.cx.metamask.io` fetch — so both branches emit a comparable envelope set.
- **Location:** `test/e2e/helpers/sentry-coverage.ts` (parse/normalize/diff, pure + unit-tested) + `test/e2e/tests/metrics/sentry-coverage.spec.ts` (capture + baseline/compare mode).
- **Protocol:** build+run on `main` (v8) → write baseline; build+run on the v10 branch → diff. Zero structural diff (modulo normalized noise) = equivalent coverage.
**3. Production staged-rollout validation (post-ship).** Canary v10, then compare the `metamask` Sentry project last-v8-release vs first-v10-release over equal windows: error volume & grouping, transaction/span volume & **perf-unit consumption** (quota — ties to #43410), tag/trace completeness (`UI Startup` / `Load Scripts` / `Background Connect` / `Setup Store` present; distributed traces still link to backend), and `tracesSampleRate` ≈ 0.75%.
## v10-specific deltas to assert explicitly (don't just trust green)
1. **Privacy-critical masking** — v10 changed *when/what* app state attaches to error events. The risk is an **unmasked sensitive field leaking**, not a missing one. Confirm `errors.spec.ts` `maskedBackgroundFields` + `sentry-get-state.test.ts` still scrub the same fields.
2. **Span serialization** — v10 broke the snap `startTrace` RPC (fixed by "Return serializable `TraceContext` from snap `startTrace` hook"). Assert snap/multichain traces serialize and complete.
3. **`propagateTraceparent` double-injection** — v10's native traceparent now overlaps the manual `consensysTracePropagation` injection. Capture outbound headers on a backend target and assert **exactly one** correct `traceparent`.
4. **Scrubbing / `beforeSend`** — confirm `rewriteReport` + `ERROR_URL_ALLOWLIST` still run (URLs rewritten, PII stripped).
## Acceptance criteria
- [ ] `sentry-coverage` parse/normalize/diff utility landed with unit tests over real captured envelope fixtures (v8 + v10).
- [ ] Capture spec records the full envelope set for the fixed flow and supports baseline + compare modes.
- [ ] v8-vs-v10 diff run produces zero structural delta (modulo normalized noise), or every delta is documented + justified.
- [ ] The four v10-specific deltas above are each explicitly asserted.
- [ ] Post-ship: prod telemetry (volume, tags, traces, sampling, perf-units) confirmed flat across the v8→v10 rollout.
## References
- `test/e2e/helpers.js` (`getEventPayloads`, `sentryRegEx`) — capture primitive to build on
- `test/e2e/tests/metrics/errors.spec.ts` + `state-snapshots/` — layer-1 snapshot oracle
- `test/e2e/tests/metrics/traces.spec.ts` — transaction-presence assertions
- `app/scripts/lib/setupSentry.js`, `sentry-get-state.ts`, `sentry-trace-propagation.ts`
- PR #42867 (the v8→v10 upgrade this verifies)
- #43410 (quota epic — the perf-unit/volume axis)
Contributor guide
Research direction
Start with test/e2e/helpers.js, especially getEventPayloads and sentryRegEx, then read the metrics specs and state-snapshots referenced in the issue. Build the pure parser, normalizer, and structural diff in test/e2e/helpers/sentry-coverage.ts with fixture tests, and add the capture flow in test/e2e/tests/metrics/sentry-coverage.spec.ts. Done means baseline and compare modes report envelope counts and deltas, the four v10-specific checks are asserted, and the v8/v10 diff is zero or documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- observability-sre, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100