opensafely-core / opensafely-core/opencodelists

Make screenshots reproducible if possible

Open
#2,658 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

deck-scrubbing
Dominant language
Python
Stars
60
Forks
16
Avg merge
4d 12h
Merged PRs (30d)
17

Description

Why are we doing this?

It may be useful to rebuild the screenshots on a scheduled basis, or add a check to validate that screenshots are up-to-date in CI checks for branches.

Checking a binary diff is easy with git diff, but maybe the screenshots change, because of, for example:

  • differences in Playwright behaviour across different operating systems
  • differences in browser rendering behaviour following a Playwright update
  • differences in test conditions (for example, dates/times — controllable, and maybe other aspects of codelist creation, such as version IDs)

Another comparison would be pixel comparison, rather than binary comparison, but again may also differ.

How will we know when it's done?

If we make the screenshot build reproducible, or we discover that this isn't possible, or it's not worth the effort.

What are we doing?

  • Changing the way the screenshots work to remove variation between runs
  • Investigating if the screenshots are reproducible
  • Maybe trying to run taking screenshot in Docker, and maybe only via a CI workflow

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the existing screenshot build, Playwright test setup, and CI workflow. Compare screenshots across repeated runs and operating-system or Playwright changes, then evaluate Docker as a controlled environment. Done means screenshot generation is reproducible, or the investigation documents why it is not feasible or worthwhile.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, playwright, python
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.