Comfy-Org / Comfy-Org/ComfyUI_frontend
Website visual baselines are container-recorded — running test:visual locally looks broken and updating it locally corrupts them
- Dominant language
- TypeScript
- Stars
- 2k
- Forks
- 704
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 512
Description
Running `pnpm --filter @comfyorg/website test:visual` locally fails 8 snapshots — Home product-cards, Home get-started, pricing-tiers, pricing-faq, at the `3-lg` and `4-xl` viewports only. They fail identically on a clean `main`, which reads like rotten baselines. It isn't.
The `-linux` baselines are recorded **inside a pinned container**:
```yaml
# .github/workflows/pr-update-website-screenshots.yaml
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.61.1-noble
```
and that workflow only fires on a label or a slash command. No PR check runs `test:visual` — the only reference to it in `.github/workflows` is the `test:visual:update` line above. So:
- The 8 local failures are a local-Chromium-vs-container mismatch, not drift on `main`.
- They gate nothing, which is why they have gone unnoticed.
- **Re-recording them locally would corrupt the baselines** for the environment that owns them. Anyone who runs `test:visual:update` on their own machine to "fix" the 8 will produce a diff that fails for everyone else, including the container.
Two things worth deciding:
1. Say this in the website README or the Playwright guidance, so the next person who runs the suite locally does not try to fix it. The trap is well disguised: the failures look real, reproduce on `main`, and the obvious remedy is the harmful one.
2. Consider whether the visual suite should run as a PR check at all. Recorded-on-demand baselines that nothing verifies will drift silently from what the pages actually look like; right now the suite can only catch a regression if someone thinks to ask it.
Contributor guide
Research direction
Start with .github/workflows/pr-update-website-screenshots.yaml and the website README or Playwright guidance; run pnpm --filter @comfyorg/website test:visual and compare its local failures with the pinned container workflow. Done means the local-baseline mismatch and safe update procedure are documented, with any PR-check decision separately resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, playwright, typescript
- Domain
- ci-cd, documentation, testing
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100