wp-media / wp-media/wp-rocket-e2e

Flaky test round 3

Open
#406 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1
Forks
1
Avg merge
3d 22h
Merged PRs (30d)
3

Description

Shared reports with AI:

scp auto-e2e-wpr-apache@57.128.123.120:~/wp-rocket-e2e/test-results-storage/2026-09-18T06-19-27-487Z/cucumber-report.html

scp auto-e2e-wpr@57.128.123.120:~/wp-rocket-e2e/test-results-storage/2026-09-18T06-06-37-576Z/cucumber-report.html

scp auto-e2e-wpr@57.128.123.120:~/wp-rocket-e2e/test-results-storage/2026-09-17T17-37-29-234Z/cucumber-report.html

scp auto-e2e-wpr@57.128.123.120:~/wp-rocket-e2e/test-results-storage/2026-09-17T11-07-14-269Z/cucumber-report.html

scp auto-e2e-wpr-apache@57.128.123.120:~/wp-rocket-e2e/test-results-storage/2026-09-17T06-24-18-962Z/cucumber-report.html

scp auto-e2e-wpr@57.128.123.120:~/wp-rocket-e2e/test-results-storage/2026-09-16T19-44-53-375Z/cucumber-report.html

scp auto-e2e-wpr@57.128.123.120:~/wp-rocket-e2e/test-results-storage/2026-09-16T14-58-20-752Z/cucumber-report.html

scp auto-e2e-wpr-apache@57.128.123.120:~/wp-rocket-e2e/test-results-storage/2026-09-16T11-23-50-714Z/cucumber-report.html

scp auto-e2e-wpr@57.128.123.120:~/wp-rocket-e2e/test-results-storage/2026-09-16T05-01-42-254Z/cucumber-report.html

Findings:
Cross-cutting root causes (affect many features)

  1. SSH connectivity to the target test site (146.59.243.71) — pure infra, not test-code flakiness
    Error: connect ECONNREFUSED 146.59.243.71:22 (12×) and Error: read ECONNRESET (9×) hit inside [hook]s across lcp-beacon-script.feature, the whole ll-css-bg-image* family, and others. This is the SSH link used to read/verify server-side state on the WP test site — separate from the 57.128.123.120 runner box. It looks like the SSH daemon on the target site drops/refuses connections under load, likely from the same CPU/resource contention already flagged in [[project_wp_rocket_e2e_activate_plugin_contention]]. Improvement: add SSH retry/backoff around that call instead of failing the whole hook on first refusal, and check whether the target site's sshd has connection-rate limits that concurrent suites are tripping.

  2. Fixed 5s/30s waits vs. a loaded shared box — the single biggest cluster

page.waitForLoadState('networkidle') / waitForSelector('a:has-text("Activate Plugin")') timing out at 30s on the "plugin is activated" step — matches the leading hypothesis already in memory ([[project_wp_rocket_e2e_activate_plugin_contention]]): concurrent auto-e2e suites (wpr, wpr-apache, bkwp, bkwp-apache) contending for CPU on one shared OVH box.
Same shape on "I am logged in" (page.waitForURL timeout) — recurs in delay-js, lcp-beacon-script, settings-export-import, notices/renewal-notice, notices/cpcss-notice, ll-css-bg-image, imagify. Login redirect is slow, not broken.
Same shape on "I must not see any error in debug.log" and "I enable all settings" — both use expect(...).toBeHidden() with a 5s timeout waiting for a loading overlay/spinner to disappear (37 + 5 occurrences, by far the largest single bucket, all in delay-js.feature and delete-plugin.feature/enable-all-features.feature).
Improvement: these all fail the same way (fixed short timeout on a box under variable load) → raise timeouts on load-dependent waits (or make them adaptive/poll-based) rather than adding retries as a band-aid. This is consistent with [[feedback_root_cause_fixes]] — fix the wait condition, don't just widen the retry net.

Feature-specific issues (real bugs, not just timing)
3. Mobile-menu step has state leakage — Error: Mobile menu is already open before attempting to open it in delay-js.feature's "expand mobile menu" scenarios. The step assumes the menu starts closed but doesn't verify/reset state first. Fix: have the step check current state and close-then-open, instead of assuming closed.

  1. settings-export-import.feature: "Nothing changed in settings '3' vs '2'" (expect(received).toBeTruthy(), always false) — 8/15 attempts fail. Genuinely flaky (mixes pass/fail across runs), pointing to a race between the settings save and the comparison read — the comparison step likely runs before the export/save has fully persisted. Fix: wait on a concrete save-confirmation signal instead of a fixed delay.

  2. imagify.feature: "Cannot save setting 'lazyload': section 'media' does not exist on the current page" (9× total, across lazyload/lazyloadYoutube/lazyloadIframes). The settings page section isn't rendered yet when the step tries to interact — a load-state race in general.ts:76, not random flake. Fix: wait for the section element itself, not just page load.

  3. settings-export-import.feature: strict-mode locator violation — getByText('Database optimization process is') matches two

    elements (running vs. complete) because the text is a prefix of both. Real selector bug, not flaky — fix: match the full string or use a more specific locator.

  4. performance-hints.feature: "Required test page 'atf-lrc-1' does not exist. Template loader plugin may have failed." (3×) — environment/fixture provisioning failure on the target site, not a script bug. Worth checking whether this correlates with the same box-contention window as #1/#2.

Bottom line
Roughly 70% of the failures trace back to two systemic causes already suspected in memory: (a) shared-box CPU contention making fixed-timeout waits ("plugin is activated", "I am logged in", debug.log check) flaky, and (b) SSH flakiness to the target site under the same load. The remaining ~30% are three concrete, fixable step-definition bugs (#3 mobile-menu state, #5 imagify section-wait, #6 strict-mode locator) plus one real race condition (#4 export/import compare) — those are worth fixing directly rather than retrying around, per your root-cause-first preference.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the shared cucumber reports and the failures in delay-js.feature, settings-export-import.feature, and performance-hints.feature. Inspect the related step definitions, including general.ts:76, then reproduce the failures under the reported shared-box conditions. Done means the identified timeout, state, persistence, locator, and fixture failures are addressed and the affected end-to-end scenarios pass reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, typescript
Domain
testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.