garris / garris/BackstopJS

Comparison issues in 4.5.0, probably arising from the upgrade to puppeteer v2

Open
#1,160 3 comments 2 reactions 0 assignees View on GitHub
needs review
Dominant language
JavaScript
Stars
7.2k
Forks
621
PR merge metrics
No merged PRs in 30d

Description

We have some 450 backstop tests that run against a library, and recently 444 of those started failing with no changes. The changes being shown were all similar to the below.

Either the fonts were being rendered differently (the screenshot taken on the right came from the most recent 4.5.0 upgrade, the left was 4.4.2):
https://ibb.co/16vt4dH

Or snapshots were being randomly cut off (sometimes what we wanted snapshot-ed did not even end up in the image; again, the image that came from 4.5.0 is to the right):
https://ibb.co/DQcCm0s

Here is an example of the configuration we are using:
```
{
"label":"Generic Components - Buttons - Button",
"url":"http://localhost:6060/#/Generic%20Components/Buttons/Button",
"selectorExpansion":true,
"selectors":[
".styleguideist-component-wrapper"
],
"misMatchThreshold":0.05,
"scrollToSelector":".styleguide-main-wrapper",
"readySelector":".styleguideist-component-wrapper",
"postInteractionWait":300,
}
```

Here are the backstop settings:

```
id: 'backstop_default',
viewports: [
{
label: 'desktop',
width: 1920,
height: 1080,
},
],
onReadyScript: 'onReady.js',
scenarios,
paths: {
bitmaps_reference: 'testing/ui/bitmaps-reference',
bitmaps_test: 'testing/ui/bitmaps-test',
engine_scripts: 'testing/ui/engine-scripts',
html_report: 'testing/ui/html-report',
ci_report: 'testing/ui/ci-report',
},
report: ['browser'],
engine: 'puppeteer',
engineOptions: {
args: ['--no-sandbox'],
},
// Helps resolve issue with misaligned screenshots
puppeteerOffscreenCaptureFix: true,
asyncCaptureLimit: 5,
asyncCompareLimit: 50,
debug: false,
debugWindow: false,
```

Each url has several blocks we want snapshot-ed separately, hence why the selector expansion. Variations to delay etc have no effect in resolving the problem.

I figured maybe the font differences are as a result of the rendering engine changing, which is fine and we could update our snapshots to match (though they no longer really appear the same as what Chrome renders). But the trimmed snaphots are a greater problem.

Previously when I had that problem, the `puppeteerOffscreenCaptureFix: true,` setting fixed it. Now, regardless of settings, the issue persists. We're not doing anything custom in our ready scripts.

At this stage I'm unsure what to change in order to make this work. Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.