codeceptjs / codeceptjs/CodeceptJS
Mochawesome/Testomatio/junit reporter: run-workers: Running of test with --suite doesn't aggregate results from all suites per browser
- Vorherrschende Sprache
- JavaScript
- Sterne
- 4.2k
- Forks
- 757
- Ø Merge
- 2 T. 9 Std.
- Gemergte PRs (30 T.)
- 16
Beschreibung
The issue is even in older versions, e.g. in 3.5.3. Retested with the latest 3.7.6 and the issue persists.
- `npm i mochawesome`
- `npm i mocha-junit-reporter`
- `npm i mocha-multi`
- Run the tests `codeceptjs run-workers 3 dev --by suite --reporter mocha-multi` (parallelization per browser and per suite). Another variant is `--suites` instead of `--by suite`, but it behaves the same way.
```
$ codeceptjs run-workers 3 dev --by suite --reporter mocha-multi
CodeceptJS v3.7.6 #StandWithUkraine
Running tests in 3 workers...
[Worker 03] ✔ SuiteB › Scenario 1 in 4ms
[Worker 03] ✔ SuiteB › Scenario 2 in 1ms
[Worker 01] ✔ SuiteA › Scenario 1 in 5ms
[Worker 01] ✔ SuiteA › Scenario 2 in 1ms
[Worker 02] ✔ SuiteA › Scenario 1 in 3ms
[Worker 02] ✔ SuiteA › Scenario 2 in 1ms
[Worker 04] ✔ SuiteB › Scenario 1 in 4ms
[Worker 04] ✔ SuiteB › Scenario 2 in 1ms
OK | 8 passed // 3s
```
**Expected result:**
- `output/console.log` contains data from both test suites
- `output/dev_chromium1/report.html` contains data from both test suites
- `output/dev_chromium1/report.json` contains data from both test suites
- `output/dev_chromium1/result.xml` contains data from both test suites
- `output/dev_firefox1/report.html` contains data from both test suites
- `output/dev_firefox1/report.json` contains data from both test suites
- `output/dev_firefox1/result.xml` contains data from both test suites
This way it works with `run-multiple` (parallelization per browser, but test suites are run in sequence) ✔️
**Actual result:**
- All the mentioned files contains data from the SuiteA only. The SuiteB is missing there 🐞 . A problem seems to be that test suites overwrite each other's results when running in threads.
---
Config `codecept.conf.ts`:
```ts
import { setHeadlessWhen, setCommonPlugins } from "@codeceptjs/configure";
// turn on headless mode when running with HEADLESS=true environment variable
// export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);
// enable all common plugins https://github.com/codeceptjs/configure#setcommonplugins
setCommonPlugins();
export const config: CodeceptJS.MainConfig = {
tests: "./*_test.ts",
output: "./output",
helpers: {
Playwright: {
browser: "chromium",
url: "http://localhost",
show: true,
},
Mochawesome: {
uniqueScreenshotNames: true,
},
},
include: {
I: "./steps_file",
},
multiple: {
dev: {
browsers: [
{
browser: "chromium",
outputName: "chromium",
},
{
browser: "firefox",
outputName: "firefox",
},
],
},
},
mocha: {
reporterOptions: {
"codeceptjs-cli-reporter": {
stdout: "-",
options: {
verbose: true,
debug: true,
steps: true,
},
},
mochawesome: {
stdout: "./output/console.log",
options: {
reportDir: "./output",
reportFilename: "report",
},
},
"mocha-junit-reporter": {
stdout: "./output/console.log",
options: {
mochaFile: "./output/result.xml",
jenkinsMode: true,
attachments: true, //add screenshot for a failed test
},
},
},
},
name: "my",
};
```
Test suites:
SuiteA_test.ts
```ts
Feature("SuiteA");
Scenario("Scenario 1", ({ I }) => {
I.say("Executing Scenario 1 in Suite A");
});
Scenario("Scenario 2", ({ I }) => {
I.say("Executing Scenario 2 in Suite A");
});
```
SuiteB_test.ts
```ts
Feature("SuiteB");
Scenario("Scenario 1", ({ I }) => {
I.say("Executing Scenario 1 in Suite B");
});
Scenario("Scenario 2", ({ I }) => {
I.say("Executing Scenario 2 in Suite B");
});
```
**Used SW:**
- CodeceptJS 3.7.6
- mochawesome 7.1.4
- mocha-junit-reporter 2.2.1
- mocha-multi 1.1.7
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.