Backstop test fails with "ENOENT: no such file or directory" if a new scenario is added
- Dominant language
- JavaScript
- Stars
- 7.2k
- Forks
- 621
- PR merge metrics
- No merged PRs in 30d
Description
Hi there!
If a new scenario is added our backstop test builds are failing in the comparison phase because backstop can't find the image for the new scenario inside the reference image folder.
The solution for the failing build is to create new reference images on that branch. This can lead to an erroneous state of the reference images on that branch if undesired changes have been made to the styling of other views.
My proposal would be to check if the reference image for a scenario is existing and marking the scenario as failed if it's not. Another solution would be to add a parameter to configure that behavior explicitly like "continue on failure". That way the failed diffs would still be marked for the existing scenarios.
Example output of the Jenkins build:
```
compare | OK: Oldpage_document_desktop.png
events.js:160
throw er; // Unhandled 'error' event
^
Error: ENOENT: no such file or directory, open '/src/src/test/backstop/data/bitmaps_reference/NewPage_document_desktop.png'
at Error (native)
```
Any thoughts or ideas on that?
Thanks for your time and effort,
Robert
Contributor guide
Assessment
This issue has not been assessed yet.