Comfy-Org / Comfy-Org/ComfyUI_frontend

[Test] Documentation troubleshooting tips for test flakiness may be insufficient

Open
#4,412 1 comment 0 reactions 0 assignees View on GitHub
area:testing documentation
Dominant language
TypeScript
Stars
2k
Forks
699
Avg merge
1d 7h
Merged PRs (30d)
490

Description

Currently, [browser_tests/README.md](https://github.com/Comfy-Org/ComfyUI_frontend/blob/main/browser_tests/README.md) states the following:

> **Screenshots vary**: Ensure your OS and browser match the reference environment (Linux)

Having had issues with font rendering myself, I felt like further instructions should be provided:

- Is there a suggested method for non-Linux users to run the tests in a Linux environment?
- I am on Linux (Ubuntu 24.04 running OpenBox), and the screenshot tests all fail due to different font rendering.
- I have tried running Playwright in a Docker container running Ubuntu 24.04 and got the same screenshot problems
- For reference, I ran docker using the following commands:
- `docker run --rm --network host -v $(pwd):/work -w /work --user $(id -u):$(id -g) mcr.microsoft.com/playwright:v1.45.1-noble npx playwright install chromium --with-deps`
- `docker run --rm --network host -v $(pwd):/work -w /work --user $(id -u):$(id -g) mcr.microsoft.com/playwright:v1.45.1-noble npx playwright test`

Online resources suggest running tests and generating screenshots in the same container should be sufficient to ensure screenshot reproducibility, and running a docker container in the same OS as the OS that the GitHub runner uses sounds close enough, so there could potentially be a rabbit hole specific to my own setup not worth sinking too much time into here (or my docker command could just be wrong). That said, I do believe something like a docker container for developers to run browser tests irrespective of their OS should be provided (or some other alternative solution).

My workaround is to simply run `npx playwright test --update-snapshots` before making any changes on my feature branch and not to commit the local snapshots. Perhaps something similar should be included in the documentation.

Lastly, I still have some failing non-screenshot tests, perhaps due to some subtle differences in my setup. This is not critical as I can just remember to ignore those failures when running the test suite. I haven't investigated why any of them fail, for now. Here's a truncated log containing failure details for reference.

Test failure log

```
Running 355 tests using 8 workers

1) [chromium] › tests/remoteWidgets.spec.ts:196:5 › Remote COMBO Widget › Refresh Behavior › refreshes options when TTL expires

Error: expect(received).not.toEqual(expected) // deep equality

Expected: not [1752139379992]

216 |
217 | const refreshedOptions = await getWidgetOptions(comfyPage, nodeName)
> 218 | expect(refreshedOptions).not.toEqual(initialOptions)
| ^
219 | })
220 |
221 | test('does not refresh when TTL is not set', async ({ comfyPage }) => {

at /home/user/coding/ComfyUI_frontend/browser_tests/tests/remoteWidgets.spec.ts:218:36

2) [chromium] › tests/sidebar/nodeLibrary.spec.ts:180:3 › Node library sidebar › Can unbookmark node (Library node bookmark)

Error: expect(received).toEqual(expected) // deep equality

- Expected - 1
+ Received + 3

- Array []
+ Array [
+ "KSamplerAdvanced",
+ ]

191 | expect(
192 | await comfyPage.getSetting('Comfy.NodeLibrary.Bookmarks.V2')
> 193 | ).toEqual([])
| ^
194 | })
195 | test('Can customize icon', async ({ comfyPage }) => {
196 | await comfyPage.setSetting('Comfy.NodeLibrary.Bookmarks.V2', ['foo/'])

at /home/user/coding/ComfyUI_frontend/browser_tests/tests/sidebar/nodeLibrary.spec.ts:193:7

3) [chromium] › tests/sidebar/workflows.spec.ts:188:3 › Workflows sidebar › Can save workflow as with same name

Error: expect(received).toEqual(expected) // deep equality

- Expected - 3
+ Received + 1

- Array [
- "workflow5.json",
- ]
+ Array []

194 | await comfyPage.menu.topbar.saveWorkflowAs('workflow5.json')
195 | await comfyPage.confirmDialog.click('overwrite')
> 196 | expect(await comfyPage.menu.workflowsTab.getOpenedWorkflowNames()).toEqual([
| ^
197 | 'workflow5.json'
198 | ])
199 | })

at /home/user/coding/ComfyUI_frontend/browser_tests/tests/sidebar/workflows.spec.ts:196:72

4) [chromium] › tests/widget.spec.ts:30:3 › Combo text widget › should refresh combo values of optional inputs

Error: expect(received).not.toEqual(expected) // deep equality

Expected: not ["Random Unique Option 1752139408.8450305", "Random Unique Option 1752139408.845039", "Random Unique Option 1752139408.8450396", "Random Unique Option 1752139408.84504", "Random Unique Option 1752139408.8450406", "Random Unique Option 1752139408.8450415", "Random Unique Option 1752139408.845042", "Random Unique Option 1752139408.8450425"]

52 |
53 | const refreshedComboValues = await getComboValues()
> 54 | expect(refreshedComboValues).not.toEqual(initialComboValues)
| ^
55 | })
56 |
57 | test('Should refresh combo values of nodes with v2 combo input spec', async ({

at /home/user/coding/ComfyUI_frontend/browser_tests/tests/widget.spec.ts:54:38

5) [chromium] › tests/widget.spec.ts:266:3 › Animated image widget › Can preview saved animated webp image

Error: Screenshot comparison failed:

24769 pixels (ratio 0.03 of all image pixels) are different.

Expected: /home/user/coding/ComfyUI_frontend/browser_tests/tests/widget.spec.ts-snapshots/animated-image-preview-saved-webp-chromium-linux.png
Received: /home/user/coding/ComfyUI_frontend/test-results/tests-widget-Animated-imag-6c3f0-w-saved-animated-webp-image-chromium/animated-image-preview-saved-webp-actual.png
Diff: /home/user/coding/ComfyUI_frontend/test-results/tests-widget-Animated-imag-6c3f0-w-saved-animated-webp-image-chromium/animated-image-preview-saved-webp-diff.png

Call log:
- expect.toHaveScreenshot(animated_image_preview_saved_webp.png) with timeout 5000ms
- verifying given screenshot expectation
- waiting for locator('#graph-canvas')
- locator resolved to
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 62765 pixels (ratio 0.07 of all image pixels) are different.
- waiting 100ms before taking screenshot
- waiting for locator('#graph-canvas')
- locator resolved to
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- 58738 pixels (ratio 0.07 of all image pixels) are different.
- waiting 250ms before taking screenshot
- waiting for locator('#graph-canvas')
- locator resolved to
- taking element screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- attempting scroll into view action
- waiting for element to be stable
- captured a stable screenshot
- 24769 pixels (ratio 0.03 of all image pixels) are different.

303 |
304 | // Expect the SaveAnimatedWEBP node to have an output preview
> 305 | await expect(comfyPage.canvas).toHaveScreenshot(
| ^
306 | 'animated_image_preview_saved_webp.png'
307 | )
308 | })

at /home/user/coding/ComfyUI_frontend/browser_tests/tests/widget.spec.ts:305:36

attachment #1: animated_image_preview_saved_webp-expected.png (image/png) ──────────────────────
browser_tests/tests/widget.spec.ts-snapshots/animated-image-preview-saved-webp-chromium-linux.png
────────────────────────────────────────────────────────────────────────────────────────────────

attachment #2: animated_image_preview_saved_webp-actual.png (image/png) ────────────────────────
test-results/tests-widget-Animated-imag-6c3f0-w-saved-animated-webp-image-chromium/animated-image-preview-saved-webp-actual.png
────────────────────────────────────────────────────────────────────────────────────────────────

attachment #3: animated_image_preview_saved_webp-diff.png (image/png) ──────────────────────────
test-results/tests-widget-Animated-imag-6c3f0-w-saved-animated-webp-image-chromium/animated-image-preview-saved-webp-diff.png
────────────────────────────────────────────────────────────────────────────────────────────────

5 failed
[chromium] › tests/remoteWidgets.spec.ts:196:5 › Remote COMBO Widget › Refresh Behavior › refreshes options when TTL expires
[chromium] › tests/sidebar/nodeLibrary.spec.ts:180:3 › Node library sidebar › Can unbookmark node (Library node bookmark)
[chromium] › tests/sidebar/workflows.spec.ts:188:3 › Workflows sidebar › Can save workflow as with same name
[chromium] › tests/widget.spec.ts:30:3 › Combo text widget › should refresh combo values of optional inputs
[chromium] › tests/widget.spec.ts:266:3 › Animated image widget › Can preview saved animated webp image
28 skipped
322 passed (2.0m)

Serving HTML report at http://localhost:45355. Press Ctrl+C to quit.
```

┆Issue is synchronized with this [Notion page](https://www.notion.so/Issue-4412-Test-Documentation-troubleshooting-tips-for-test-flakiness-may-be-insufficient-22c6d73d36508186be9df8cca6eb50bb) by [Unito](https://www.unito.io)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.