internetarchive / internetarchive/openlibrary

Run the end-to-end suite in CI

Open
#13,605 1 comment 0 reactions 1 assignee Claimed by @RayBB View on GitHub
Affects: CI/CD Lead: @lokesh Module: Docker Needs: Breakdown Priority: 3 Theme: Development Theme: Testing Type: Epic Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

## Problem / Opportunity

The Playwright suite in `tests/e2e/` — 12 specs covering home, search, work, edition, author, subjects, login, My Books, `ol-button` forms, `ol-carousel`, plus an axe-core a11y pass — runs nowhere automatically. No CI job, no pre-commit hook. It runs when a contributor remembers to run it.

That is how four of those specs quietly drifted out of step with the app within weeks of landing: home asserted an account link that moved into the hamburger drawer, login asserted a post-login landing page the route no longer picks, search gated on a client-side page title and so skipped itself in about a third of runs, and the a11y spec scanned while the lists dropper still had a loading placeholder in its `

    `. Nobody noticed, because nothing ran them. #13555 fixes those specs and reduces setup to `make e2e-up` / `make test-e2e`, but it doesn't stop the same drift from happening again. A suite that isn't run protects nothing — it just accumulates false failures until someone deletes it.

    **Success:** a PR that breaks the header, search results, or a Lit component turns a check red on GitHub, and the suite's state is visible without anyone having to remember it exists.

    ## Proposal

    Add `.github/workflows/e2e_tests.yml`, mirroring what `make e2e-up` does locally: bring up the stack, rebuild assets from the PR's tree, ensure Solr has data, install Chromium, run `make test-e2e`.

    The config work is already done in #13555 — `retries: 2` when `CI` is set, `forbidOnly`, `trace: 'on-first-retry'`, screenshot and video retained on failure. This issue is only the workflow.

    **This is not a copy of `javascript_tests.yml`.** Both existing workflows run on the host with `uv` and `npm`; neither starts Docker. This job would be the first to run the real stack (`db memcached mockservices infobase covers solr web fast_web`), and that is where the cost and the traps are:

    - **The image.** `compose.yaml` has no build section for `oldev`, so CI has to `docker pull openlibrary/olbase:latest` (public, rebuilt weekly by `olbase.yaml`) and then `docker build -f docker/Dockerfile.oldev`. This will dominate the job's runtime; layer caching is worth getting right before anything else.
    - **Solr data.** The search, subjects and author specs skip themselves against an empty index. `make reindex-solr` curls `osp_totals.db` from archive.org and indexes the seed DB. Dev Solr runs a 128m heap (`compose.override.yaml:77`) and has been seen to exit 134 under write bursts, so an unattended reindex on a runner is a genuine flake source, not a formality.
    - **Skips read exactly like passes.** 8 of 60 tests skip by design today (7 in `visual.spec.ts`, 1 `test.fixme`). The data-dependent specs skip silently on top of that. If the job doesn't assert which tests are allowed to skip, a green check can mean the browser never loaded a search result.
    - **Assets.** `static/build` is a volume seeded from the image, built from **master**. Without `make e2e-assets` the browser tests master's bundles and a spec covering the PR's own JS passes without running it.
    - **`visual.spec.ts` must stay out**, permanently — full-page screenshots differ with platform font rendering, so `OL_VISUAL` stays unset in CI.

    ### Suggested shape

    Start where the signal is cheap and promote once it's proven:

    1. Land it as `workflow_dispatch` + nightly on master first. Watch it for a week or two — the question that matters is whether the stack comes up reliably on a runner, and that's answerable without blocking anyone's PR.
    2. Then gate PRs, scoped by `paths` to what can actually break it: `openlibrary/templates/**`, `openlibrary/plugins/openlibrary/js/**`, `openlibrary/components/**`, `static/css/**`, `tests/e2e/**`, `Makefile`, `compose*.yaml`, and the workflow itself. If the full run can't get under a few minutes, gate on `@smoke` only and leave the rest nightly.
    3. Leave it out of `make test`. That target runs on any checkout; this needs a stack and a populated index.

    Implementation Details (for maintainers)

    #### Related files
    * `.github/workflows/javascript_tests.yml` — closest existing workflow to model the Node/npm caching on
    * `Makefile` — `e2e-stack`, `e2e-assets`, `e2e-index`, `test-e2e` (added in #13555); the job should call these, not re-implement them
    * `playwright.config.ts` — CI retries, `forbidOnly`, trace/screenshot/video already set
    * `tests/e2e/README.md` — the "Where these run" section currently reads "Nowhere automatic, yet"
    * `docker/Dockerfile.oldev`, `compose.yaml`, `compose.override.yaml`

    #### Requirements Checklist
    * [ ] Workflow runs the suite against a real stack and is green on master
    * [ ] Demonstrably red on a deliberate breakage (rename `#header-bar` and watch it fail)
    * [ ] No silent green: the job fails if a Solr-dependent spec skips
    * [ ] `playwright-report/` and `test-results/` uploaded as artifacts on failure
    * [ ] `visual.spec.ts` stays skipped (`OL_VISUAL` unset)
    * [ ] `tests/e2e/README.md` and `CONTRIBUTING.md` updated to say where the suite now runs

    #### Open questions
    * PR-gating or nightly to start — depends entirely on how long the image pull and index take on a runner.
    * Can a populated Solr index be cached between runs (snapshot the `solr-data` volume, or bake an indexed image) rather than reindexing every time?
    * Does the a11y spec gate PRs on day one, or report only? An axe violation is easy to introduce from a change that looks unrelated to the page it fires on.

    #### Stakeholders
    *

    ## Related

    * #13555 — makes the suite one command and fixes the drifted specs; this issue is its stated follow-up, and should land first so the CI job starts from a known-green suite
    * #12885 — the issue that created the suite
    * #13007 / #13251 — the axe-core integration the a11y spec came from
    * #1833 — cross-browser testing epic; this job runs Chromium only

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.