garris / garris/BackstopJS

CircleCI: false positive result

Open
#1,145 8 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
7.2k
Forks
621
PR merge metrics
No merged PRs in 30d

Description

I'm trying to integrate BackstopJS with the CircleCI and having a problem with receiving a false-positive result.
- I have 104 tests, 395 occurrences of `compare | ERROR { requireSameDimensions: false, size: isDifferent, content: 0.14%, threshold: 1e-7% }: some-image.png` and tests are passing successfully.
- I'm removing all of the tests except one or two - they fail(as expected)

there is no difference between first and second run, except the number of test cases.

**config:**
```json
{
"id": "some_id",
"onReadyScript": "som_js_script.js",
"viewports": [
{
"name": "1680",
"width": 1680,
"height": 4000
},
{
"name": "1920",
"width": 1920,
"height": 4000
},
{
"name": "2560",
"width": 2560,
"height": 4000
}
],
"scenarios": [
{
"label": "scaffolding",
"url": "path_for_the_first_test.html",
"selectors": ["document"],
"misMatchThreshold": 0.0000001,
"delay": 1000,
"viewports": [
{
"label": "Default",
"width": 1920,
"height": 4000
}
]
},
],
"paths": {
"bitmaps_reference": "backstop_data/bitmaps_reference",
"bitmaps_test": "backstop_data/bitmaps_test",
"engine_scripts": "backstop_data/engine_scripts/puppet/",
"html_report": "backstop_data/html_report",
"ci_report": "backstop_data/ci_report"
},
"report": ["CI"],
"engineOptions": {
"args": ["--no-sandbox"]
},
"engine": "puppeteer",
"asyncCaptureLimit": 5
}
```

and also, as test completes successfully - there is no artifacts.

**ENV:**
All tests are running on the CircleCI
node: circleci/node:8.15.0-jessie

**P.S.** I'm currently trying to kind of bisect search through all of my tests to maybe find some test that causes this, so it will be easier to find the reason for that bug. So will add more info later

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.