garris / garris/BackstopJS

Significant degradation in performance using Docker

Open
#1,573 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7.2k
Forks
621
PR merge metrics
No merged PRs in 30d

Description

Has anyone else struggled with a significant degradation in performance when running through Docker? I am looking to run a large number of scenarios (320) against a local running instance of an application. I am using three viewport sizes. When I run on my M1 Mac, the run takes approximatley 3 minutes...

```
> test
> backstop test --config="backstop.config.js"
BackstopJS v6.3.23
Loading config: [...]/packages/visual-testing/backstop.config.js
COMMAND | Executing core for "test"
createBitmaps | Selected 320 of 320 scenarios.
Creating Browser
No Playwright browser specified, assuming Chromium.
Browser created
Browser Console Log 0: BackstopTools have been installed.
[...]
report | Test completed...
report | 960 Passed
report | 0 Failed
report | Writing json report
report | Writing browser report
report | Resources copied
report | Wrote Json report to: [...]/packages/visual-testing/data/reports/json_report/jsonReport.json
report | Resources copied
report | Copied json report to: [...]/packages/visual-testing/data/evidence/20240613-084244/report.json
report | Copied jsonp report to: [...]/packages/visual-testing/data/reports/html_report/config.js
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening data/reports/html_report/index.html
COMMAND | Command "report" successfully executed in [13.696s]
COMMAND | Command "test" successfully executed in [161.644s]
```

To run my tests I am running `backstop test --config="backstop.config.js"`.

Because I need the tests to run on my local development environment and in GitHub Actions I have implementing running in docker following similar to [this](https://github.com/garris/BackstopJS?tab=readme-ov-file#using-docker-for-testing-across-different-environments).

When I run using Docker (`backstop test --config="backstop.config.js" --docker`) and it gives me an outcome like this...

```
> test
> backstop test --config="backstop.config.js" --docker
BackstopJS v6.3.23
Loading config: [...]/packages/visual-testing/backstop.config.js
COMMAND | Executing core for "test"
Delegating command to Docker... docker run --rm [...] -w /app/packages/visual-testing --name tests -v [...]/apps:/app/apps:ro -v [...]/packages/visual-testing:/app/packages/visual-testing:ro backstopjs/backstopjs:6.3.23 test "--config=backstop.config.js" "--moby"
BackstopJS v6.3.23
Loading config: /app/packages/visual-testing/backstop.config.js
COMMAND | Executing core for "test"
createBitmaps | Selected 320 of 320 scenarios.
Creating Browser
No Playwright browser specified, assuming Chromium.
Browser created
Browser Console Log 0: BackstopTools have been installed.
[...]
report | Test completed...
report | 954 Passed
report | 6 Failed
report | Writing json report
report | Writing browser report
report | Resources copied
report | Wrote Json report to: /app/packages/visual-testing/data/reports/json_report/jsonReport.json
report | Resources copied
report | Copied json report to: /app/packages/visual-testing/data/evidence/20240613-074636/report.json
report | Copied jsonp report to: /app/packages/visual-testing/data/reports/html_report/config.js
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
openReport | Remote not found. Opening data/reports/html_report/index.html
[...]
COMMAND | Executing core for "openReport"
openReport | Attempting to ping
COMMAND | Command "test" ended with an error after [4540.916s]
COMMAND | Error: docker run --rm [...] -w /app/packages/visual-testing --name tests -v [...]/apps:/app/apps:ro -v [...]/packages/visual-testing:/app/packages/visual-testing:ro backstopjs/backstopjs:6.3.23 test "--config=backstop.config.js" "--moby" returned 1
at ChildProcess. ([...]/node_modules/backstopjs/core/util/runDocker.js:68:18)
at ChildProcess.emit (node:events:527:28)
at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
openReport | Remote not found. Opening data/reports/html_report/index.html
npm
ERR! Lifecycle script `test` failed with error:
npm ERR! Error: command failed
npm ERR! in workspace: visual-testing
npm ERR! at location: [...]/packages/visual-testing
Error in script: 1
```
(the failures are minor discrepancies which show why I need to get my project running in Docker)

My config is dynamically built from JSON files but looks something like this...

```
module.exports = {
asyncCaptureLimit: 15,
asyncCompareLimit: 50,
ci: {
format: "junit",
testSuiteName: "backstopJS",
},
debug: process.env.DEBUG === "true",
debugWindow: process.env.DEBUG === "true",
dockerCommandTemplate: getDockerRunCommand(),
engine: "playwright",
engineOptions: {
args: ["--no-sandbox", "--disable-setuid-sandbox"],
browsersTypes: ["chromium"],
gotoParameters: {
waitUntil: "networkidle0",
},
headless: "new",
ignoreHTzTPSErrors: false,
},
fileNameTemplate: "{scenarioLabel}_{viewportLabel}",
id: "gfe-storybook",
misMatchThreshold: 20,
onBeforeScript: "playwright/onBefore.js",
onReadyScript: "playwright/onReady.js",
paths,
report,
requireSameDimensions: false,
scenarioDefaults: {
delay: 0,
selectorExpansion: true,
},
scenarios: getScenarios(),
viewports: [
{
label: "desktop",
width: 1920,
height: 1080,
},
{
label: "tablet",
width: 1024,
height: 768,
},
{
label: "phone",
width: 320,
height: 480,
},
],
};

```

My project is part of a mono repo so the mounts look a bit strange and I have redacted the full file paths.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.