internetarchive / internetarchive/openlibrary
Run the end-to-end suite in CI
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
## Problem / Opportunity
The Playwright suite in `tests/e2e/` — 12 specs covering home, search, work, edition, author, subjects, login, My Books, `ol-button` forms, `ol-carousel`, plus an axe-core a11y pass — runs nowhere automatically. No CI job, no pre-commit hook. It runs when a contributor remembers to run it.
That is how four of those specs quietly drifted out of step with the app within weeks of landing: home asserted an account link that moved into the hamburger drawer, login asserted a post-login landing page the route no longer picks, search gated on a client-side page title and so skipped itself in about a third of runs, and the a11y spec scanned while the lists dropper still had a loading placeholder in its `
- `. Nobody noticed, because nothing ran them. #13555 fixes those specs and reduces setup to `make e2e-up` / `make test-e2e`, but it doesn't stop the same drift from happening again. A suite that isn't run protects nothing — it just accumulates false failures until someone deletes it.
**Success:** a PR that breaks the header, search results, or a Lit component turns a check red on GitHub, and the suite's state is visible without anyone having to remember it exists.
## Proposal
Add `.github/workflows/e2e_tests.yml`, mirroring what `make e2e-up` does locally: bring up the stack, rebuild assets from the PR's tree, ensure Solr has data, install Chromium, run `make test-e2e`.
The config work is already done in #13555 — `retries: 2` when `CI` is set, `forbidOnly`, `trace: 'on-first-retry'`, screenshot and video retained on failure. This issue is only the workflow.
**This is not a copy of `javascript_tests.yml`.** Both existing workflows run on the host with `uv` and `npm`; neither starts Docker. This job would be the first to run the real stack (`db memcached mockservices infobase covers solr web fast_web`), and that is where the cost and the traps are:
- **The image.** `compose.yaml` has no build section for `oldev`, so CI has to `docker pull openlibrary/olbase:latest` (public, rebuilt weekly by `olbase.yaml`) and then `docker build -f docker/Dockerfile.oldev`. This will dominate the job's runtime; layer caching is worth getting right before anything else.
- **Solr data.** The search, subjects and author specs skip themselves against an empty index. `make reindex-solr` curls `osp_totals.db` from archive.org and indexes the seed DB. Dev Solr runs a 128m heap (`compose.override.yaml:77`) and has been seen to exit 134 under write bursts, so an unattended reindex on a runner is a genuine flake source, not a formality.
- **Skips read exactly like passes.** 8 of 60 tests skip by design today (7 in `visual.spec.ts`, 1 `test.fixme`). The data-dependent specs skip silently on top of that. If the job doesn't assert which tests are allowed to skip, a green check can mean the browser never loaded a search result.
- **Assets.** `static/build` is a volume seeded from the image, built from **master**. Without `make e2e-assets` the browser tests master's bundles and a spec covering the PR's own JS passes without running it.
- **`visual.spec.ts` must stay out**, permanently — full-page screenshots differ with platform font rendering, so `OL_VISUAL` stays unset in CI.
### Suggested shape
Start where the signal is cheap and promote once it's proven:
1. Land it as `workflow_dispatch` + nightly on master first. Watch it for a week or two — the question that matters is whether the stack comes up reliably on a runner, and that's answerable without blocking anyone's PR.
2. Then gate PRs, scoped by `paths` to what can actually break it: `openlibrary/templates/**`, `openlibrary/plugins/openlibrary/js/**`, `openlibrary/components/**`, `static/css/**`, `tests/e2e/**`, `Makefile`, `compose*.yaml`, and the workflow itself. If the full run can't get under a few minutes, gate on `@smoke` only and leave the rest nightly.
3. Leave it out of `make test`. That target runs on any checkout; this needs a stack and a populated index.
Implementation Details (for maintainers)
#### Related files
* `.github/workflows/javascript_tests.yml` — closest existing workflow to model the Node/npm caching on
* `Makefile` — `e2e-stack`, `e2e-assets`, `e2e-index`, `test-e2e` (added in #13555); the job should call these, not re-implement them
* `playwright.config.ts` — CI retries, `forbidOnly`, trace/screenshot/video already set
* `tests/e2e/README.md` — the "Where these run" section currently reads "Nowhere automatic, yet"
* `docker/Dockerfile.oldev`, `compose.yaml`, `compose.override.yaml`
#### Requirements Checklist
* [ ] Workflow runs the suite against a real stack and is green on master
* [ ] Demonstrably red on a deliberate breakage (rename `#header-bar` and watch it fail)
* [ ] No silent green: the job fails if a Solr-dependent spec skips
* [ ] `playwright-report/` and `test-results/` uploaded as artifacts on failure
* [ ] `visual.spec.ts` stays skipped (`OL_VISUAL` unset)
* [ ] `tests/e2e/README.md` and `CONTRIBUTING.md` updated to say where the suite now runs
#### Open questions
* PR-gating or nightly to start — depends entirely on how long the image pull and index take on a runner.
* Can a populated Solr index be cached between runs (snapshot the `solr-data` volume, or bake an indexed image) rather than reindexing every time?
* Does the a11y spec gate PRs on day one, or report only? An axe violation is easy to introduce from a change that looks unrelated to the page it fires on.
#### Stakeholders
*
## Related
* #13555 — makes the suite one command and fixes the drifted specs; this issue is its stated follow-up, and should land first so the CI job starts from a known-green suite
* #12885 — the issue that created the suite
* #13007 / #13251 — the axe-core integration the a11y spec came from
* #1833 — cross-browser testing epic; this job runs Chromium only
Contributor guide
Assessment
This issue has not been assessed yet.