Comparison issues in 4.5.0, probably arising from the upgrade to puppeteer v2
- Dominant language
- JavaScript
- Stars
- 7.2k
- Forks
- 621
- PR merge metrics
- No merged PRs in 30d
Description
We have some 450 backstop tests that run against a library, and recently 444 of those started failing with no changes. The changes being shown were all similar to the below.
Either the fonts were being rendered differently (the screenshot taken on the right came from the most recent 4.5.0 upgrade, the left was 4.4.2):
https://ibb.co/16vt4dH
Or snapshots were being randomly cut off (sometimes what we wanted snapshot-ed did not even end up in the image; again, the image that came from 4.5.0 is to the right):
https://ibb.co/DQcCm0s
Here is an example of the configuration we are using:
```
{
"label":"Generic Components - Buttons - Button",
"url":"http://localhost:6060/#/Generic%20Components/Buttons/Button",
"selectorExpansion":true,
"selectors":[
".styleguideist-component-wrapper"
],
"misMatchThreshold":0.05,
"scrollToSelector":".styleguide-main-wrapper",
"readySelector":".styleguideist-component-wrapper",
"postInteractionWait":300,
}
```
Here are the backstop settings:
```
id: 'backstop_default',
viewports: [
{
label: 'desktop',
width: 1920,
height: 1080,
},
],
onReadyScript: 'onReady.js',
scenarios,
paths: {
bitmaps_reference: 'testing/ui/bitmaps-reference',
bitmaps_test: 'testing/ui/bitmaps-test',
engine_scripts: 'testing/ui/engine-scripts',
html_report: 'testing/ui/html-report',
ci_report: 'testing/ui/ci-report',
},
report: ['browser'],
engine: 'puppeteer',
engineOptions: {
args: ['--no-sandbox'],
},
// Helps resolve issue with misaligned screenshots
puppeteerOffscreenCaptureFix: true,
asyncCaptureLimit: 5,
asyncCompareLimit: 50,
debug: false,
debugWindow: false,
```
Each url has several blocks we want snapshot-ed separately, hence why the selector expansion. Variations to delay etc have no effect in resolving the problem.
I figured maybe the font differences are as a result of the rendering engine changing, which is fine and we could update our snapshots to match (though they no longer really appear the same as what Chrome renders). But the trimmed snaphots are a greater problem.
Previously when I had that problem, the `puppeteerOffscreenCaptureFix: true,` setting fixed it. Now, regardless of settings, the issue persists. We're not doing anything custom in our ready scripts.
At this stage I'm unsure what to change in order to make this work. Thank you!
Contributor guide
Assessment
This issue has not been assessed yet.