CircleCI: false positive result
- Dominant language
- JavaScript
- Stars
- 7.2k
- Forks
- 621
- PR merge metrics
- No merged PRs in 30d
Description
I'm trying to integrate BackstopJS with the CircleCI and having a problem with receiving a false-positive result.
- I have 104 tests, 395 occurrences of `compare | ERROR { requireSameDimensions: false, size: isDifferent, content: 0.14%, threshold: 1e-7% }: some-image.png` and tests are passing successfully.
- I'm removing all of the tests except one or two - they fail(as expected)
there is no difference between first and second run, except the number of test cases.
**config:**
```json
{
"id": "some_id",
"onReadyScript": "som_js_script.js",
"viewports": [
{
"name": "1680",
"width": 1680,
"height": 4000
},
{
"name": "1920",
"width": 1920,
"height": 4000
},
{
"name": "2560",
"width": 2560,
"height": 4000
}
],
"scenarios": [
{
"label": "scaffolding",
"url": "path_for_the_first_test.html",
"selectors": ["document"],
"misMatchThreshold": 0.0000001,
"delay": 1000,
"viewports": [
{
"label": "Default",
"width": 1920,
"height": 4000
}
]
},
],
"paths": {
"bitmaps_reference": "backstop_data/bitmaps_reference",
"bitmaps_test": "backstop_data/bitmaps_test",
"engine_scripts": "backstop_data/engine_scripts/puppet/",
"html_report": "backstop_data/html_report",
"ci_report": "backstop_data/ci_report"
},
"report": ["CI"],
"engineOptions": {
"args": ["--no-sandbox"]
},
"engine": "puppeteer",
"asyncCaptureLimit": 5
}
```
and also, as test completes successfully - there is no artifacts.
**ENV:**
All tests are running on the CircleCI
node: circleci/node:8.15.0-jessie
**P.S.** I'm currently trying to kind of bisect search through all of my tests to maybe find some test that causes this, so it will be easier to find the reason for that bug. So will add more info later
Contributor guide
Assessment
This issue has not been assessed yet.