garris / garris/BackstopJS

Replacing chromy with puppeteer (originally: Failing with no images in report, and no screenshot for test taken)

Open
#724 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7.2k
Forks
621
PR merge metrics
No merged PRs in 30d

Description

I'm having some issues with randomly getting failed tests where the error report does not show any images;

Console log:
![backstop-error_a-01_console](https://user-images.githubusercontent.com/1295067/38468598-fa0347f2-3b48-11e8-825d-3a24e12ba074.png)

Error report:
![backstop-error_a-02_report](https://user-images.githubusercontent.com/1295067/38468599-fe529a4c-3b48-11e8-90dd-ede49e16a398.png)

Chrome Dev tools inspection of error report:
![backstop-error_a-03_inspect-report](https://user-images.githubusercontent.com/1295067/38468601-032e2c0c-3b49-11e8-8246-1745295bdd21.png)

Error report, show diffs clicked (broken image link):
![backstop-error_a-04_show-diffs](https://user-images.githubusercontent.com/1295067/38468602-06114544-3b49-11e8-884a-e2c5cfc04f37.png)

Chrome Dev tools inspection of show diffs error report (notice img src attrs):
![backstop-error_a-05_show-diffs-inspect](https://user-images.githubusercontent.com/1295067/38468604-0c204796-3b49-11e8-80cc-0e6d03c05826.png)

The failed compare ("Home-Welcome 02 Toolbar (tablet-landscape)", according to console log) does have a reference image but in the test folder no image for this test exists.

I had the issue when asyncCaptureLimit was set to 5, I have experimented with lowering it but still these kind of fails keeps happening randomly.

It seems more likely to happen when I'm running the batch of all my scenarios over running only a few. But I just now had a case when I filtered out 3 scenarios and set the asyncCaptureLimit to 1 and not a single screenshot was taken, so not even a folder was created in the bitmaps_test folder. This happens repeatedly right now.

Any suggestions on what I can do? I'm eager to introduce the tool to my colleagues and run backstop within our CI but due to this issue it does not feel stable enough.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.