garris / garris/BackstopJS

Report "Mismatch errors found" only when regression difference between reference and test

Open
#1,241 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7.2k
Forks
621
PR merge metrics
No merged PRs in 30d

Description

It looks like the open report function works when there is no difference between `reference` and `test`. However it fails when there is a difference.

Output when comparing `reference` and `test` and there **is no visual regression difference**:

```
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening backstop_data/html_report/index.html
COMMAND | Command "report" successfully executed in [1.144s]
COMMAND | Command "test" successfully executed in [1.745s]
```

Output when comparing `reference` and `test` and there **is a visual regression difference**:

```
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening backstop_data/html_report/index.html
report | *** Mismatch errors found ***
COMMAND | Command "report" ended with an error after [1.167s]
COMMAND | Error: Mismatch errors found.
at /app/static/node_modules/backstopjs/core/command/report.js:189:17
at processTicksAndRejections (internal/process/task_queues.js:97:5)
COMMAND | Command "test" ended with an error after [1.95s]
COMMAND | Error: Mismatch errors found.
at /app/static/node_modules/backstopjs/core/command/report.js:189:17
at processTicksAndRejections (internal/process/task_queues.js:97:5)
```

I am using a slightly simplified version of the default `backstop.json` file:

```
{
"id": "default",
"viewports": [
{
"label": "Small",
"width": 320,
"height": 480
}
],
"scenarios": [
{
"label": "Homepage",
"url": "http://localhost:8081/index.html",
"referenceUrl": "",
"readyEvent": "",
"readySelector": "",
"delay": 0,
"hideSelectors": [],
"removeSelectors": [],
"hoverSelector": "",
"clickSelector": "",
"postInteractionWait": 0,
"selectors": [],
"selectorExpansion": true,
"expect": 0,
"misMatchThreshold" : 0.1,
"requireSameDimensions": true
}
],
"paths": {
"bitmaps_reference": "backstop_data/bitmaps_reference",
"bitmaps_test": "backstop_data/bitmaps_test",
"engine_scripts": "backstop_data/engine_scripts",
"html_report": "backstop_data/html_report",
"ci_report": "backstop_data/ci_report"
},
"report": ["browser"],
"engine": "puppeteer",
"engineOptions": {
"args": ["--no-sandbox"]
},
"asyncCaptureLimit": 5,
"asyncCompareLimit": 50,
"debug": false,
"debugWindow": false
}
```

I am running a localhost of static HTML files which I server like this:

```
# Run local webserver and detach.
php -S localhost:8081 -t ./dist/ &
```

# Recreate issue

## 1. Run a `backstop reference`:

```
$ backstop reference
BackstopJS v5.0.6
Loading config: /app/static/backstop.json

COMMAND | Executing core for "reference"
clean | backstop_data/bitmaps_reference was cleaned.
createBitmaps | Selected 1 of 1 scenarios.
CREATING NEW REFERENCE FILE
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55572 Accepted
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55572 [200]: (null) /index.html
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55572 Closing
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55574 Accepted
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55574 [200]: (null) /assets/fonts/roboto.woff2
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55576 Accepted
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55576 [200]: (null) /assets/images/master.svg
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55574 Closing
[Fri Oct 9 08:06:16 2020] 127.0.0.1:55576 Closing
Browser Console Log 0: JSHandle:BackstopTools have been installed.
x Close Browser

Run `$ backstop test` to generate diff report.

COMMAND | Command "reference" successfully executed in [0.686s]
```

# 2. Run `backstop test` of exact same project files

This step works fine and does not find any difference in the project files.

```
$ backstop test
BackstopJS v5.0.6
Loading config: /app/static/backstop.json

COMMAND | Executing core for "test"
createBitmaps | Selected 1 of 1 scenarios.
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55580 Accepted
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55580 [200]: (null) /index.html
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55580 Closing
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55582 Accepted
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55584 Accepted
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55582 [200]: (null) /assets/fonts/roboto.woff2
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55584 [200]: (null) /assets/images/master.svg
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55582 Closing
[Fri Oct 9 08:06:22 2020] 127.0.0.1:55584 Closing
Browser Console Log 0: JSHandle:BackstopTools have been installed.
x Close Browser
COMMAND | Executing core for "report"
compare | OK: Homepage default_Homepage_0_document_0_Small.png
report | Test completed...
report | 1 Passed
report | 0 Failed
report | Writing browser report
report | Resources copied
report | Copied jsonp report to: /app/static/backstop_data/html_report/config.js
report | Copied json report to: /app/static/backstop_data/bitmaps_test/20201009-080622/report.json
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening backstop_data/html_report/index.html
COMMAND | Command "report" successfully executed in [1.144s]
COMMAND | Command "test" successfully executed in [1.745s]
$
```

# 3. I make a visual change the project file and run backstop test

```
$ backstop test
BackstopJS v5.0.6
Loading config: /app/static/backstop.json

COMMAND | Executing core for "test"
createBitmaps | Selected 1 of 1 scenarios.
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55590 Accepted
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55590 [200]: (null) /index.html
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55590 Closing
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55592 Accepted
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55594 Accepted
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55592 [200]: (null) /assets/fonts/roboto.woff2
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55594 [200]: (null) /assets/images/master.svg
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55592 Closing
[Fri Oct 9 08:06:38 2020] 127.0.0.1:55594 Closing
Browser Console Log 0: JSHandle:BackstopTools have been installed.
x Close Browser
COMMAND | Executing core for "report"
See: /app/static/backstop_data/bitmaps_test/20201009-080638/failed_diff_default_Homepage_0_document_0_Small.png
compare | ERROR { requireSameDimensions: true, size: isDifferent, content: 0.38%, threshold: 0.1% }: Homepage default_Homepage_0_document_0_Small.png
report | Test completed...
report | 0 Passed
report | 1 Failed
report | Writing browser report
report | Resources copied
report | Copied jsonp report to: /app/static/backstop_data/html_report/config.js
report | Copied json report to: /app/static/backstop_data/bitmaps_test/20201009-080638/report.json
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening backstop_data/html_report/index.html
report | *** Mismatch errors found ***
COMMAND | Command "report" ended with an error after [1.167s]
COMMAND | Error: Mismatch errors found.
at /app/static/node_modules/backstopjs/core/command/report.js:189:17
at processTicksAndRejections (internal/process/task_queues.js:97:5)
COMMAND | Command "test" ended with an error after [1.95s]
COMMAND | Error: Mismatch errors found.
at /app/static/node_modules/backstopjs/core/command/report.js:189:17
at processTicksAndRejections (internal/process/task_queues.js:97:5)
$
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.