MetaMask / MetaMask/metamask-extension

[P2] Some mock suites for e2e and benchmarks are parallel implementations that drift silently

Open
#45,449 1 comment 0 reactions 0 assignees View on GitHub
INVALID-ISSUE-TEMPLATE Sev2-normal ta-needs-engineer-escalation ta-triaged team-qa
Dominant language
TypeScript
Stars
13.2k
Forks
5.6k
Avg merge
2d 5h
Merged PRs (30d)
451

Description

**Parent Epic:** [#6944 Performance Quality Gates](https://github.com/MetaMask/MetaMask-planning/issues/6944) → [#7182 Benchmarks: Reliability](https://github.com/MetaMask/MetaMask-planning/issues/7182) → [#7181 Benchmark Harness Fixes](https://github.com/MetaMask/MetaMask-planning/issues/7181)
**File:** `test/e2e/mock-e2e.js`, `test/e2e/benchmarks/mocks/performance-mocks.ts`, `test/e2e/benchmarks/mocks/mock-responses.ts`
**Size:** M | **Hours:** ~8-12

---

## Problem

The e2e suite and the benchmark suite maintain separate mock implementations of the same upstream services. A fix to one does not reach the other, and the divergence is silent — both suites pass, and the gap only shows up as a timing anomaly weeks later.

This has now happened twice on the same code path.

### The instance

[#43818 globalize non-EVM snap discovery mocks](https://github.com/MetaMask/metamask-extension/pull/43818) added `setupDefaultNonEvmDiscoveryMocks` to `test/e2e/mock-e2e.js`, mocking Solana's `getSignaturesForAddress`. [#43961 complete Solana discovery RPC mocks](https://github.com/MetaMask/metamask-extension/pull/43961) completed it, adding well-formed results for the fifteen JSON-RPC methods the Solana snap calls during discovery — the unmocked ones were falling through to a generic empty-200 catch-all whose malformed bodies throw inside the snap and restart discovery, a ~516-request retry storm.

The benchmark suite has its own mock implementation and never loaded that helper. Read at `0350a018`: `test/e2e/mock-e2e.js:261` defines `setupDefaultNonEvmDiscoveryMocks`, covering `getSignaturesForAddress` directly plus fourteen more methods via a loop over `SOLANA_DISCOVERY_RPC_RESULTS` (lines 173–204). `test/e2e/benchmarks/mocks/performance-mocks.ts` explicitly matches six of those — `getBalance`, `getAccountInfo`, `getLatestBlockhash`, `getFeeForMessage`, `getTokenAccountsByOwner`, `getSignaturesForAddress` — and routes the rest to `solanaCatchAllResponse` ([`mock-responses.ts:510-519`](https://github.com/MetaMask/metamask-extension/blob/0350a01878e0d5e0eb1553b18ca7d653b4157cbb/test/e2e/benchmarks/mocks/mock-responses.ts#L510-L519)), which returns `{ context, value: null }` — the same wrong-shape condition, reproduced. [#45443 port the completed Solana discovery mocks into the benchmark suite](https://github.com/MetaMask/metamask-extension/pull/45443) ports the result map across, closing the instance; it is open as of 2026-09-02.

The cost was not the port. It was that [#45266 onboarding benchmarks are bimodal](https://github.com/MetaMask/metamask-extension/issues/45266) recorded "**#43961 did not fix it**" on the strength of a sample taken 26 hours after that PR merged, and the real explanation — a second implementation that never received the fix — took a separate investigation to find.

### Why it stays expensive

- **Both suites pass while diverged.** There is no assertion anywhere that the two cover the same request surface, so divergence has no detector.
- **The benchmark suite's failure mode is quieter.** In e2e a bad mock usually surfaces as a flaky assertion. In benchmarks it surfaces as a number, and a number that is merely wrong looks exactly like a number that is correct.
- **They are not obviously related.** At `0350a018` the only cross-references between the two trees are a type-only `import type { MockedEndpoint }` in `swap.ts:10` and three imports of the unrelated `mock-e2e-pass-through` helper. Neither tree imports the other's mock registrations or fixtures, so nothing in review flags that a change to one leaves the other stale.

---

## Solution

Options, roughly in increasing cost. This ticket does not pick one — the choice depends on how much of the benchmark suite's calibrated-delay behavior has to be preserved, which is the part that genuinely differs.

1. **Share the fixtures, keep the registration separate.** Extract the response *shapes* (`SOLANA_DISCOVERY_RPC_RESULTS` and its peers) into one module both suites import. The benchmark suite keeps its own delays and priorities. Smallest change, and it fixes the class of divergence seen here — wrong-shaped bodies — without touching timing.
2. **Share the registration, parameterize the delays.** One `setupNonEvmDiscoveryMocks(server, { delays })`, called by both. Removes more duplication and more freedom.
3. **A coverage assertion.** A test that enumerates the methods each suite matches and fails when one covers a request the other does not. Catches future divergence without merging anything, and composes with (1).

(1) plus (3) is the likely answer: dedupe what must not differ, and detect it when it drifts anyway.

---

## Acceptance Criteria

- [ ] Response shapes for non-EVM discovery live in one place, imported by both suites
- [ ] The benchmark suite keeps its calibrated per-method delays
- [ ] A test fails when one suite matches a request surface the other does not
- [ ] Adding a method to the shared set requires no edit in the second suite

---

## Labels

`team-extension-platform`, `area-testSuite`

---

## Dependencies

**Related:** [#45443 port the completed Solana discovery mocks into the benchmark suite](https://github.com/MetaMask/metamask-extension/pull/45443) (ports the current instance, open), [#45266 onboarding benchmarks are bimodal](https://github.com/MetaMask/metamask-extension/issues/45266) (the slow path this misdiagnosed), [#43961 complete Solana discovery RPC mocks](https://github.com/MetaMask/metamask-extension/pull/43961), [#43958 complete non-EVM discovery mocks](https://github.com/MetaMask/metamask-extension/issues/43958), [#43818 globalize non-EVM discovery mocks](https://github.com/MetaMask/metamask-extension/pull/43818)

Contributor guide

Open the contributing guide

Research direction

Start by reading test/e2e/mock-e2e.js, test/e2e/benchmarks/mocks/performance-mocks.ts, and test/e2e/benchmarks/mocks/mock-responses.ts, focusing on SOLANA_DISCOVERY_RPC_RESULTS and the benchmark delays. Decide how to share response shapes while preserving calibrated delays, then add coverage that detects mismatched request surfaces. Done means both suites use one shared response set and the acceptance criteria pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
testing
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.