Comfy-Org / Comfy-Org/ComfyUI_frontend
[Test] Regressions are not being caught by Playwright
- Dominant language
- TypeScript
- Stars
- 2k
- Forks
- 699
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 490
Description
## Problem
We are missing regressions due to Playwright's fuzzy-matching logic.
From observation, it seems "few pixels + drastic colour change" is weighted higher than "many pixels + subtle colour change". This can cause a shift in color / brightness of a decent-size rectangle to be completely ignored.
### Example
A test had been passing despite having a somewhat significant difference.
1. The code was "just" within the acceptable margin of drift of Playwright's fuzzy-matching - passing, just barely
2. An extremely minor update was done - a visually-imperceptible, sub-pixel shift was made
3. The test was now "just over" the fuzzy-matching limit, due to the anti-aliased edges of some rectangles changing colour ever so slightly
**Result**:
- It appeared that a regression had occurred in what was unrelated to the actual change being made.
- It was not this component, but it wasn't time well spent.
- We have no indication when the regression occurred, because the test that should of caught it never did.
### Expectation -> Actual
This is the expectation vs actual BEFORE the PR - which _should_ have been failing. This was verified by just running the tests manually.
->
The group is missing - inside white outline rectangle.
## Solution
Fine-tune Playwright's fuzzy-matching values.
┆Issue is synchronized with this [Notion page](https://www.notion.so/Issue-5899-Test-Adjust-Playwright-fuzzy-values-acceptable-drift-2816d73d36508176bc5edfdddf7c207d) by [Unito](https://www.unito.io)
Contributor guide
Assessment
This issue has not been assessed yet.