google / google/site-kit-wp

Fix flaky E2E tests.

Open
#12,262 1 comment 0 reactions 1 assignee Claimed by @eugene-manuilov View on GitHub
Next Up P1 Team S Type: Bug Type: Infrastructure
Dominant language
JavaScript
Stars
1.4k
Forks
383
Avg merge
4d 14h
Merged PRs (30d)
77

Description

## Feature Description

Now that #12258 is closed, the remaining flakiness in the Puppeteer E2E suite is concentrated in two tests. Sampling the seven most recent failing `develop` runs (June 8 to 11), every failure except two one-off blips came from one or both of the tests below.

**1. `tests/e2e/specs/auth/reauthentication-notice.test.js`**

`Reauthentication needed admin notice › redirects to analytics setup after reauth if the flow has been previously interrupted`

Fails with `Execution context was destroyed, most likely because of a navigation.` across all three WordPress matrix jobs:

- https://github.com/google/site-kit-wp/actions/runs/27370570577 (June 11, WordPress latest)
- https://github.com/google/site-kit-wp/actions/runs/27275733490 (June 10, WordPress 5.2.21)
- https://github.com/google/site-kit-wp/actions/runs/27201509567 (June 9, WordPress latest)
- https://github.com/google/site-kit-wp/actions/runs/27198746476 (June 9, WordPress nightly)
- https://github.com/google/site-kit-wp/actions/runs/27153482507 (June 8, WordPress latest)

**2. `tests/e2e/specs/modules/analytics/write-scope-requests.test.js`**

`Analytics write scope requests › prompts for additional permissions during a new Analytics web data stream creation if the user has not granted the Analytics edit scope`

Fails with a console error assertion: `Google Site Kit API Error method:POST datapoint:set-google-tag-id-mismatch type:modules identifier:analytics-4 error:"The response is not a valid JSON response."`, always on the WordPress 5.2.21 + AMP job:

- https://github.com/google/site-kit-wp/actions/runs/27370570577 (June 11)
- https://github.com/google/site-kit-wp/actions/runs/27245948643 (June 10)
- https://github.com/google/site-kit-wp/actions/runs/27153482507 (June 8)
- https://github.com/google/site-kit-wp/actions/runs/27122166083 (June 8)

This test was previously stabilised in #10570; the failure mode has since changed from a `TimeoutError` to the console error above.

We may wish to create separate issues for the tests depending on how complicated fixing them turns out to be.

---------------

_Do not alter or remove anything below. The following sections will be managed by moderators only._

## Acceptance criteria

* The `redirects to analytics setup after reauth if the flow has been previously interrupted` test in `tests/e2e/specs/auth/reauthentication-notice.test.js` passes consistently in CI.
* The `prompts for additional permissions during a new Analytics web data stream creation if the user has not granted the Analytics edit scope` test in `tests/e2e/specs/modules/analytics/write-scope-requests.test.js` passes consistently in CI.
* Any other intermittent E2E failures observed during the investigation are captured as separate issues.

## Implementation Brief

This is an investigation ticket. The candidates below are starting points, not confirmed root causes.

* [ ] Update file `tests/e2e/specs/auth/reauthentication-notice.test.js`: investigate the `Execution context was destroyed` failure. Candidates: the final `toClick` on the reauthentication notice triggers the intercepted OAuth redirect chain while the click assertion is still evaluating in the page context, so wrap the click and `page.waitForNavigation()` in a `Promise.all()`; the closing `toMatchElement` runs after a single `waitForNavigation()` but the OAuth callback flow involves more than one redirect, so wait for `.googlesitekit-setup-module--analytics` with `page.waitForSelector()` instead.
* [ ] Update file `tests/e2e/specs/modules/analytics/write-scope-requests.test.js`: investigate why the `set-google-tag-id-mismatch` POST escapes the request interception and reaches the live REST API, where no route is mocked and WordPress 5.2.21 returns a non-JSON response. Candidates: the request fires around the post-setup dashboard navigation where interception responses can be dropped, so move the datapoint from the generic fallback loop in the `interceptions` map to an explicit branch, or register a server-side mock for the datapoint in `tests/e2e/plugins/module-setup-analytics.php`.
* [ ] Confirm stability by re-running the `E2E Tests` workflow on the PR branch several times, e.g. five consecutive green runs.
* [ ] If either test needs a substantial rework rather than a targeted fix, create a follow-up issue and link it here.

### Test Coverage

* No new tests; the deliverable is the two existing E2E tests passing consistently in CI.

## QA Brief

*

## Changelog entry

*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.