Comfy-Org / Comfy-Org/ComfyUI_frontend

Website visual baselines are container-recorded — running test:visual locally looks broken and updating it locally corrupts them

Open
#14,793 1 comment 1 reaction 0 assignees View on GitHub
area:testing documentation
Dominant language
TypeScript
Stars
2k
Forks
704
Avg merge
1d 8h
Merged PRs (30d)
512

Description

Running `pnpm --filter @comfyorg/website test:visual` locally fails 8 snapshots — Home product-cards, Home get-started, pricing-tiers, pricing-faq, at the `3-lg` and `4-xl` viewports only. They fail identically on a clean `main`, which reads like rotten baselines. It isn't.

The `-linux` baselines are recorded **inside a pinned container**:

```yaml
# .github/workflows/pr-update-website-screenshots.yaml
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.61.1-noble
```

and that workflow only fires on a label or a slash command. No PR check runs `test:visual` — the only reference to it in `.github/workflows` is the `test:visual:update` line above. So:

- The 8 local failures are a local-Chromium-vs-container mismatch, not drift on `main`.
- They gate nothing, which is why they have gone unnoticed.
- **Re-recording them locally would corrupt the baselines** for the environment that owns them. Anyone who runs `test:visual:update` on their own machine to "fix" the 8 will produce a diff that fails for everyone else, including the container.

Two things worth deciding:

1. Say this in the website README or the Playwright guidance, so the next person who runs the suite locally does not try to fix it. The trap is well disguised: the failures look real, reproduce on `main`, and the obvious remedy is the harmful one.
2. Consider whether the visual suite should run as a PR check at all. Recorded-on-demand baselines that nothing verifies will drift silently from what the pages actually look like; right now the suite can only catch a regression if someone thinks to ask it.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/pr-update-website-screenshots.yaml and the website README or Playwright guidance; run pnpm --filter @comfyorg/website test:visual and compare its local failures with the pinned container workflow. Done means the local-baseline mismatch and safe update procedure are documented, with any PR-check decision separately resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, playwright, typescript
Domain
ci-cd, documentation, testing
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.