Issue with pass/fail tests at the same time
- Dominant language
- JavaScript
- Stars
- 7.2k
- Forks
- 621
- PR merge metrics
- No merged PRs in 30d
Description
Hi @garris,
I got issue with a test in my CI which is passing and failing at the same time which worries me. More interestingly it will fully pass it I re-run it several times!
Here is the logs:
```
console.log node_modules/backstopjs/core/util/makeConfig.js:39
BackstopJS uses a passed object as config
console.log node_modules/backstopjs/core/util/logger.js:63
COMMAND | Executing core for `test`
console.log node_modules/backstopjs/core/util/runCasper.js:31
Running CasperJS with: [ '/drone/src/git.prod.xxx.local/web-platform/header/node_modules/backstopjs/capture/genBitmaps.js',
'--captureConfigFileName=/drone/src/git.prod.xxx.local/web-platform/header/capture/cc194dbd24313344f89908e28d5f55a171c0c8f7.json' ]
console.error node_modules/backstopjs/core/util/runCasper.js:44
CasperJS: 2017-06-28T13:11:16 [CRITICAL] QNetworkReplyImpl: backend error: caching was enabled after some bytes had been written
console.log node_modules/backstopjs/core/util/runCasper.js:40
CasperJS: Warning: Accessing PropTypes via the main React package is deprecated, and will be removed in React v16.0. Use the latest available v15.* prop-types package from npm instead. For info on usage, compatibility, migration and more, see https://fb.me/prop-types-docs
console.log node_modules/backstopjs/core/util/runCasper.js:40
CasperJS: Warning: Accessing PropTypes via the main React package is deprecated, and will be removed in React v16.0. Use the latest available v15.* prop-types package from npm instead. For info on usage, compatibility, migration and more, see https://fb.me/prop-types-docs
console.log node_modules/backstopjs/core/util/runCasper.js:40
CasperJS: Ready event received.
console.log node_modules/backstopjs/core/util/runCasper.js:40
CasperJS: Current location is http://localhost:3000/
console.log node_modules/backstopjs/core/util/runCasper.js:40
CasperJS: Capturing screenshots for phone (320x480)
console.log node_modules/backstopjs/core/util/runCasper.js:40
CasperJS: Comparison config file updated.
console.log node_modules/backstopjs/core/util/createBitmaps.js:66
Bitmap file generation completed.
console.log node_modules/backstopjs/core/util/logger.js:63
COMMAND | Executing core for `report`
console.log node_modules/backstopjs/core/util/logger.js:63
compare | ERROR reference image not found/drone/src/git.prod.xxx.local/web-platform/header/src/visualTests/backstop_data/bitmaps_reference/prod_test_defaultConfigs_1_header-root__n1_0_phone.png: defaultConfigs prod_test_defaultConfigs_1_header-root__n1_0_phone.png
console.log node_modules/backstopjs/core/util/logger.js:63
compare | OK: defaultConfigs prod_test_defaultConfigs_0_header-root_0_phone.png
console.log node_modules/backstopjs/core/util/logger.js:63
report | Test completed...
console.log node_modules/backstopjs/core/util/logger.js:63
report | 1 Passed
console.log node_modules/backstopjs/core/util/logger.js:63
report | 1 Failed
console.log node_modules/backstopjs/core/util/logger.js:63
report | Writing jUnit Report
console.log node_modules/backstopjs/core/util/logger.js:63
report | Writing browser report
console.log node_modules/backstopjs/core/util/logger.js:63
report | jUnit report written to: ../backstop_data/ci_report/xunit.xml
console.log node_modules/backstopjs/core/util/logger.js:63
report | Browser reported copied
console.log node_modules/backstopjs/core/util/logger.js:63
report | Copied configuration to: /drone/src/git.prod.xxx.local/web-platform/backstop_data/html_report/config.js
console.log node_modules/backstopjs/core/util/logger.js:63
COMMAND | Executing core for `openReport`
console.log node_modules/backstopjs/core/util/logger.js:63
openReport | Opening report.
console.log node_modules/backstopjs/core/util/logger.js:63
openReport | An error occured while opening report in the default browser.
console.log node_modules/backstopjs/core/util/logger.js:63
COMMAND | Command `openReport` sucessfully executed
console.log node_modules/backstopjs/core/util/logger.js:63
report | *** Mismatch errors found ***
console.log node_modules/backstopjs/core/util/logger.js:63
report | For a detailed report run `backstop openReport`
console.log node_modules/backstopjs/core/util/logger.js:63
COMMAND | Command `report` ended with an error
console.log node_modules/backstopjs/core/util/logger.js:63
COMMAND | Error: Mismatch errors found.
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/backstopjs/core/command/report.js:113:17
at propagateAslWrapper (/drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/index.js:468:23)
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/index.js:505:70
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/glue.js:188:31
at propagateAslWrapper (/drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/index.js:468:23)
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/index.js:505:70
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/glue.js:188:31
at process._tickDomainCallback (internal/process/next_tick.js:129:7)
console.log node_modules/backstopjs/core/util/logger.js:63
COMMAND | Command `test` ended with an error
console.log node_modules/backstopjs/core/util/logger.js:63
COMMAND | Error: Mismatch errors found.
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/backstopjs/core/command/report.js:113:17
at propagateAslWrapper (/drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/index.js:468:23)
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/index.js:505:70
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/glue.js:188:31
at propagateAslWrapper (/drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/index.js:468:23)
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/index.js:505:70
at /drone/src/git.prod.xxx.local/web-platform/header/node_modules/async-listener/glue.js:188:31
at process._tickDomainCallback (internal/process/next_tick.js:129:7)
```
And config:
```
{
id: "1",
viewports: [
{
name: 'phone',
width: 320,
height: 480,
},
],
scenarios: [
{
label: 'defaultConfigs',
url: 'http://localhost:3000/',
hideSelectors: [],
removeSelectors: [],
selectorExpansion: true,
selectors: ['#header-root'],
readyEvent: null,
delay: 500,
misMatchThreshold: 0.1,
requireSameDimensions: true,
},
],
paths: {
bitmaps_reference: 'src/visualTests/backstop_data/bitmaps_reference',
bitmaps_test: '../backstop_data/bitmaps_test',
html_report: '../backstop_data/html_report',
ci_report: '../backstop_data/ci_report',
},
ci,
casperFlags: [],
engine: 'phantomjs',
report: ['browser', 'CI'],
debug: false,
}
```
Any idea how that would be possible?
Contributor guide
Assessment
This issue has not been assessed yet.