Comfy-Org / Comfy-Org/ComfyUI_frontend

[Test] Regressions are not being caught by Playwright

Open
#5,899 2 comments 0 reactions 0 assignees View on GitHub
area:testing developer experience
Dominant language
TypeScript
Stars
2k
Forks
699
Avg merge
1d 7h
Merged PRs (30d)
490

Description

## Problem

We are missing regressions due to Playwright's fuzzy-matching logic.

From observation, it seems "few pixels + drastic colour change" is weighted higher than "many pixels + subtle colour change". This can cause a shift in color / brightness of a decent-size rectangle to be completely ignored.

### Example

A test had been passing despite having a somewhat significant difference.

1. The code was "just" within the acceptable margin of drift of Playwright's fuzzy-matching - passing, just barely
2. An extremely minor update was done - a visually-imperceptible, sub-pixel shift was made
3. The test was now "just over" the fuzzy-matching limit, due to the anti-aliased edges of some rectangles changing colour ever so slightly

**Result**:
- It appeared that a regression had occurred in what was unrelated to the actual change being made.
- It was not this component, but it wasn't time well spent.
- We have no indication when the regression occurred, because the test that should of caught it never did.

### Expectation -> Actual

This is the expectation vs actual BEFORE the PR - which _should_ have been failing. This was verified by just running the tests manually.

Image -> Image

The group is missing - inside white outline rectangle.

## Solution

Fine-tune Playwright's fuzzy-matching values.

┆Issue is synchronized with this [Notion page](https://www.notion.so/Issue-5899-Test-Adjust-Playwright-fuzzy-values-acceptable-drift-2816d73d36508176bc5edfdddf7c207d) by [Unito](https://www.unito.io)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.