garris / garris/BackstopJS

Output format JPG not compatible with Lemur fallback images

Open
#1,438 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7.2k
Forks
621
PR merge metrics
No merged PRs in 30d

Description

Hello everybody!

We're heavily using the `outputFormat: "jpg"`-option in large projects, as this helps us keeping the repository (reference files are stored in LFS) smaller. I think we might have discovered a problem with the fallback images (e.g. `'/capture/resources/unexpectedErrorSm.png'`). These are PNGs, regardless the `outputFormat` option. So whenever some of these fallback images are stored as a reference, the following test-run results in an exception like:

```
See: /src/tests/backstop/bitmaps_test/20220929-064330/failed_diff_xxx.jpg
/usr/local/lib/node_modules/backstopjs/core/util/compare/store-failed-diff.js:20
fs.writeFileSync(failedDiffFilename, data.getDiffImageAsJPEG(85));
^

TypeError: data.getDiffImageAsJPEG is not a function
at module.exports (/usr/local/lib/node_modules/backstopjs/core/util/compare/store-failed-diff.js:20:47)
at /usr/local/lib/node_modules/backstopjs/core/util/compare/compare.js:21:14
at processTicksAndRejections (node:internal/process/task_queues:96:5)
```

Even this might be acceptable in the end, right now this error is catched by the test runner and test execution continues/ends with success. When running tests as part of the CI pipeline everything seems pretty ok, and no errors are reported. This error can only be discovered by reading the test entire output, what - at least I think so - usually doesn't happen in an automated ci-env.

IMHO the error should cause the test run to exit with an error code. Having different fallback/error images relying on the `outputFormat` could help too 😄 . I'm interested in your opinion on it before trying to contribute a possible solution.

Contributor guide

Open the contributing guide

Research direction

Start by reading core/util/compare/store-failed-diff.js and core/util/compare/compare.js, then reproduce a run using outputFormat: "jpg" with a PNG fallback image such as /capture/resources/unexpectedErrorSm.png. Trace how the getDiffImageAsJPEG error is caught and determine the expected failure behavior for the test runner and CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.