Report "Mismatch errors found" only when regression difference between reference and test
- Dominant language
- JavaScript
- Stars
- 7.2k
- Forks
- 621
- PR merge metrics
- No merged PRs in 30d
Description
It looks like the open report function works when there is no difference between `reference` and `test`. However it fails when there is a difference.
Output when comparing `reference` and `test` and there **is no visual regression difference**:
```
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening backstop_data/html_report/index.html
COMMAND | Command "report" successfully executed in [1.144s]
COMMAND | Command "test" successfully executed in [1.745s]
```
Output when comparing `reference` and `test` and there **is a visual regression difference**:
```
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening backstop_data/html_report/index.html
report | *** Mismatch errors found ***
COMMAND | Command "report" ended with an error after [1.167s]
COMMAND | Error: Mismatch errors found.
at /app/static/node_modules/backstopjs/core/command/report.js:189:17
at processTicksAndRejections (internal/process/task_queues.js:97:5)
COMMAND | Command "test" ended with an error after [1.95s]
COMMAND | Error: Mismatch errors found.
at /app/static/node_modules/backstopjs/core/command/report.js:189:17
at processTicksAndRejections (internal/process/task_queues.js:97:5)
```
I am using a slightly simplified version of the default `backstop.json` file:
```
{
"id": "default",
"viewports": [
{
"label": "Small",
"width": 320,
"height": 480
}
],
"scenarios": [
{
"label": "Homepage",
"url": "http://localhost:8081/index.html",
"referenceUrl": "",
"readyEvent": "",
"readySelector": "",
"delay": 0,
"hideSelectors": [],
"removeSelectors": [],
"hoverSelector": "",
"clickSelector": "",
"postInteractionWait": 0,
"selectors": [],
"selectorExpansion": true,
"expect": 0,
"misMatchThreshold" : 0.1,
"requireSameDimensions": true
}
],
"paths": {
"bitmaps_reference": "backstop_data/bitmaps_reference",
"bitmaps_test": "backstop_data/bitmaps_test",
"engine_scripts": "backstop_data/engine_scripts",
"html_report": "backstop_data/html_report",
"ci_report": "backstop_data/ci_report"
},
"report": ["browser"],
"engine": "puppeteer",
"engineOptions": {
"args": ["--no-sandbox"]
},
"asyncCaptureLimit": 5,
"asyncCompareLimit": 50,
"debug": false,
"debugWindow": false
}
```
I am running a localhost of static HTML files which I server like this:
```
# Run local webserver and detach.
php -S localhost:8081 -t ./dist/ &
```
# Recreate issue
## 1. Run a `backstop reference`:
```
$ backstop reference
BackstopJS v5.0.6
Loading config: /app/static/backstop.json
COMMAND | Executing core for "reference"
clean | backstop_data/bitmaps_reference was cleaned.
createBitmaps | Selected 1 of 1 scenarios.
CREATING NEW REFERENCE FILE
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55572 Accepted
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55572 [200]: (null) /index.html
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55572 Closing
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55574 Accepted
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55574 [200]: (null) /assets/fonts/roboto.woff2
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55576 Accepted
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55576 [200]: (null) /assets/images/master.svg
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55574 Closing
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55576 Closing
Browser Console Log 0: JSHandle:BackstopTools have been installed.
x Close Browser
Run `$ backstop test` to generate diff report.
COMMAND | Command "reference" successfully executed in [0.686s]
```
# 2. Run `backstop test` of exact same project files
This step works fine and does not find any difference in the project files.
```
$ backstop test
BackstopJS v5.0.6
Loading config: /app/static/backstop.json
COMMAND | Executing core for "test"
createBitmaps | Selected 1 of 1 scenarios.
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55580 Accepted
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55580 [200]: (null) /index.html
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55580 Closing
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55582 Accepted
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55584 Accepted
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55582 [200]: (null) /assets/fonts/roboto.woff2
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55584 [200]: (null) /assets/images/master.svg
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55582 Closing
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55584 Closing
Browser Console Log 0: JSHandle:BackstopTools have been installed.
x Close Browser
COMMAND | Executing core for "report"
compare | OK: Homepage default_Homepage_0_document_0_Small.png
report | Test completed...
report | 1 Passed
report | 0 Failed
report | Writing browser report
report | Resources copied
report | Copied jsonp report to: /app/static/backstop_data/html_report/config.js
report | Copied json report to: /app/static/backstop_data/bitmaps_test/20201009-080622/report.json
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening backstop_data/html_report/index.html
COMMAND | Command "report" successfully executed in [1.144s]
COMMAND | Command "test" successfully executed in [1.745s]
$
```
# 3. I make a visual change the project file and run backstop test
```
$ backstop test
BackstopJS v5.0.6
Loading config: /app/static/backstop.json
COMMAND | Executing core for "test"
createBitmaps | Selected 1 of 1 scenarios.
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55590 Accepted
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55590 [200]: (null) /index.html
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55590 Closing
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55592 Accepted
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55594 Accepted
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55592 [200]: (null) /assets/fonts/roboto.woff2
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55594 [200]: (null) /assets/images/master.svg
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55592 Closing
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55594 Closing
Browser Console Log 0: JSHandle:BackstopTools have been installed.
x Close Browser
COMMAND | Executing core for "report"
See: /app/static/backstop_data/bitmaps_test/20201009-080638/failed_diff_default_Homepage_0_document_0_Small.png
compare | ERROR { requireSameDimensions: true, size: isDifferent, content: 0.38%, threshold: 0.1% }: Homepage default_Homepage_0_document_0_Small.png
report | Test completed...
report | 0 Passed
report | 1 Failed
report | Writing browser report
report | Resources copied
report | Copied jsonp report to: /app/static/backstop_data/html_report/config.js
report | Copied json report to: /app/static/backstop_data/bitmaps_test/20201009-080638/report.json
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening backstop_data/html_report/index.html
report | *** Mismatch errors found ***
COMMAND | Command "report" ended with an error after [1.167s]
COMMAND | Error: Mismatch errors found.
at /app/static/node_modules/backstopjs/core/command/report.js:189:17
at processTicksAndRejections (internal/process/task_queues.js:97:5)
COMMAND | Command "test" ended with an error after [1.95s]
COMMAND | Error: Mismatch errors found.
at /app/static/node_modules/backstopjs/core/command/report.js:189:17
at processTicksAndRejections (internal/process/task_queues.js:97:5)
$
```
Contributor guide
Assessment
This issue has not been assessed yet.