"Test" rendering is different than reference even when pointed to the same site
- Dominant language
- JavaScript
- Stars
- 7.2k
- Forks
- 621
- PR merge metrics
- No merged PRs in 30d
Description
I'm trying to build backstop testing into our gitlab cicd pipelines and I've come across an oddity that I can not debug: when running backstop from the backstopjs docker image, the test **always** fails at least one scenario even if the reference and test scenarios point to the same domain (ie reference = identity.missouri.edu, test = identity.missouri.edu). And more confusing is that it is not the same scenario(s) that fails during each run.
Running the exact same commands locally produces a successful test.
I'm using the docker image backstopjs/backstopjs:latest
backstop version is 5.3.0
After starting the image (which ships with node v10), I updated node to v15.12.0 to match what our developers are working with locally.
The project contains a package-lock.js. I then do `npm ci` to ensure we get a clean install of all node dependencies.
I have verified that node, backstop, Chromium, etc are all running the exact same versions in the docker image as local.
When it fails, the "test" rendering appears to be shifted down by 1px and the text all appears to have adopted different tracking. It is VERY similar to the image you have here: https://github.com/garris/BackstopJS#using-docker-for-testing-across-different-environments except that I'm generating both the reference bitmaps and the test bitmaps in the _same_ docker container, one right after the other.
backstop_config.js is below. I've tried it with both the `docker: true` and that property removed.
Happy to provide any additional information you need to debug this one.
```
/**
* @file
* Backstop configuration.
*/
const args = require('minimist')(process.argv.slice(2));
const {scenarios,storagepath} = require(`./backstop_scenarios`);
const path = require('path');
const site = args.site ? args.site+'/' : '';
module.exports = {
id: 'visual_test',
viewports: [
{
label: 'phone',
width: 320,
height: 480,
},
{
label: 'tablet-portrait',
width: 700,
height: 1024,
},
{
label: 'tablet-landscape',
width: 1024,
height: 768,
},
{
label: 'desktop',
width: 1280,
height: 1024,
},
{
label: 'desktop-wide',
width: 2000,
height: 1024,
},
],
onBeforeScript: 'puppet/onBefore.js',
onReadyScript: 'puppet/onReady.js',
scenarios: scenarios,
paths: {
bitmaps_reference: path.resolve(storagepath,'backstop_data',site,'bitmaps_reference'),
bitmaps_test: path.resolve(storagepath,'backstop_data',site,'bitmaps_test'),
engine_scripts: path.resolve(__dirname,'backstop_data','engine_scripts'),
html_report: path.resolve(storagepath,'backstop_data',site,'html_report'),
ci_report: path.resolve(storagepath,'backstop_data',site,'ci_report'),
},
report: ['CI'],
engine: 'puppeteer',
engineOptions: {
args: ['--no-sandbox'],
},
asyncCaptureLimit: 1,
asyncCompareLimit: 50,
debug: false,
debugWindow: false,
docker: true,
};
```
Contributor guide
Assessment
This issue has not been assessed yet.