Fix flaky E2E tests.
- Dominant language
- JavaScript
- Stars
- 1.4k
- Forks
- 383
- Avg merge
- 4d 14h
- Merged PRs (30d)
- 77
Description
## Feature Description
Now that #12258 is closed, the remaining flakiness in the Puppeteer E2E suite is concentrated in two tests. Sampling the seven most recent failing `develop` runs (June 8 to 11), every failure except two one-off blips came from one or both of the tests below.
**1. `tests/e2e/specs/auth/reauthentication-notice.test.js`**
`Reauthentication needed admin notice › redirects to analytics setup after reauth if the flow has been previously interrupted`
Fails with `Execution context was destroyed, most likely because of a navigation.` across all three WordPress matrix jobs:
- https://github.com/google/site-kit-wp/actions/runs/27370570577 (June 11, WordPress latest)
- https://github.com/google/site-kit-wp/actions/runs/27275733490 (June 10, WordPress 5.2.21)
- https://github.com/google/site-kit-wp/actions/runs/27201509567 (June 9, WordPress latest)
- https://github.com/google/site-kit-wp/actions/runs/27198746476 (June 9, WordPress nightly)
- https://github.com/google/site-kit-wp/actions/runs/27153482507 (June 8, WordPress latest)
**2. `tests/e2e/specs/modules/analytics/write-scope-requests.test.js`**
`Analytics write scope requests › prompts for additional permissions during a new Analytics web data stream creation if the user has not granted the Analytics edit scope`
Fails with a console error assertion: `Google Site Kit API Error method:POST datapoint:set-google-tag-id-mismatch type:modules identifier:analytics-4 error:"The response is not a valid JSON response."`, always on the WordPress 5.2.21 + AMP job:
- https://github.com/google/site-kit-wp/actions/runs/27370570577 (June 11)
- https://github.com/google/site-kit-wp/actions/runs/27245948643 (June 10)
- https://github.com/google/site-kit-wp/actions/runs/27153482507 (June 8)
- https://github.com/google/site-kit-wp/actions/runs/27122166083 (June 8)
This test was previously stabilised in #10570; the failure mode has since changed from a `TimeoutError` to the console error above.
We may wish to create separate issues for the tests depending on how complicated fixing them turns out to be.
---------------
_Do not alter or remove anything below. The following sections will be managed by moderators only._
## Acceptance criteria
* The `redirects to analytics setup after reauth if the flow has been previously interrupted` test in `tests/e2e/specs/auth/reauthentication-notice.test.js` passes consistently in CI.
* The `prompts for additional permissions during a new Analytics web data stream creation if the user has not granted the Analytics edit scope` test in `tests/e2e/specs/modules/analytics/write-scope-requests.test.js` passes consistently in CI.
* Any other intermittent E2E failures observed during the investigation are captured as separate issues.
## Implementation Brief
This is an investigation ticket. The candidates below are starting points, not confirmed root causes.
* [ ] Update file `tests/e2e/specs/auth/reauthentication-notice.test.js`: investigate the `Execution context was destroyed` failure. Candidates: the final `toClick` on the reauthentication notice triggers the intercepted OAuth redirect chain while the click assertion is still evaluating in the page context, so wrap the click and `page.waitForNavigation()` in a `Promise.all()`; the closing `toMatchElement` runs after a single `waitForNavigation()` but the OAuth callback flow involves more than one redirect, so wait for `.googlesitekit-setup-module--analytics` with `page.waitForSelector()` instead.
* [ ] Update file `tests/e2e/specs/modules/analytics/write-scope-requests.test.js`: investigate why the `set-google-tag-id-mismatch` POST escapes the request interception and reaches the live REST API, where no route is mocked and WordPress 5.2.21 returns a non-JSON response. Candidates: the request fires around the post-setup dashboard navigation where interception responses can be dropped, so move the datapoint from the generic fallback loop in the `interceptions` map to an explicit branch, or register a server-side mock for the datapoint in `tests/e2e/plugins/module-setup-analytics.php`.
* [ ] Confirm stability by re-running the `E2E Tests` workflow on the PR branch several times, e.g. five consecutive green runs.
* [ ] If either test needs a substantial rework rather than a targeted fix, create a follow-up issue and link it here.
### Test Coverage
* No new tests; the deliverable is the two existing E2E tests passing consistently in CI.
## QA Brief
*
## Changelog entry
*
Contributor guide
Assessment
This issue has not been assessed yet.