codeceptjs / codeceptjs/CodeceptJS

Mochawesome/Testomatio/junit reporter: run-workers: Running of test with --suite doesn't aggregate results from all suites per browser

Đang mở
#5,411 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
JavaScript
Star
4.2k
Fork
757
Merge trung bình
2 ngày 9 giờ
Pull request đã merge (30 ngày)
16

Mô tả

The issue is even in older versions, e.g. in 3.5.3. Retested with the latest 3.7.6 and the issue persists.

- `npm i mochawesome`
- `npm i mocha-junit-reporter`
- `npm i mocha-multi`
- Run the tests `codeceptjs run-workers 3 dev --by suite --reporter mocha-multi` (parallelization per browser and per suite). Another variant is `--suites` instead of `--by suite`, but it behaves the same way.

```
$ codeceptjs run-workers 3 dev --by suite --reporter mocha-multi
CodeceptJS v3.7.6 #StandWithUkraine
Running tests in 3 workers...

[Worker 03] ✔ SuiteB › Scenario 1 in 4ms
[Worker 03] ✔ SuiteB › Scenario 2 in 1ms
[Worker 01] ✔ SuiteA › Scenario 1 in 5ms
[Worker 01] ✔ SuiteA › Scenario 2 in 1ms
[Worker 02] ✔ SuiteA › Scenario 1 in 3ms
[Worker 02] ✔ SuiteA › Scenario 2 in 1ms
[Worker 04] ✔ SuiteB › Scenario 1 in 4ms
[Worker 04] ✔ SuiteB › Scenario 2 in 1ms

OK | 8 passed // 3s
```

**Expected result:**

- `output/console.log` contains data from both test suites
- `output/dev_chromium1/report.html` contains data from both test suites
- `output/dev_chromium1/report.json` contains data from both test suites
- `output/dev_chromium1/result.xml` contains data from both test suites
- `output/dev_firefox1/report.html` contains data from both test suites
- `output/dev_firefox1/report.json` contains data from both test suites
- `output/dev_firefox1/result.xml` contains data from both test suites

This way it works with `run-multiple` (parallelization per browser, but test suites are run in sequence) ✔️

**Actual result:**

- All the mentioned files contains data from the SuiteA only. The SuiteB is missing there 🐞 . A problem seems to be that test suites overwrite each other's results when running in threads.

---

Config `codecept.conf.ts`:
```ts
import { setHeadlessWhen, setCommonPlugins } from "@codeceptjs/configure";
// turn on headless mode when running with HEADLESS=true environment variable
// export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);

// enable all common plugins https://github.com/codeceptjs/configure#setcommonplugins
setCommonPlugins();

export const config: CodeceptJS.MainConfig = {
tests: "./*_test.ts",
output: "./output",
helpers: {
Playwright: {
browser: "chromium",
url: "http://localhost",
show: true,
},
Mochawesome: {
uniqueScreenshotNames: true,
},
},
include: {
I: "./steps_file",
},
multiple: {
dev: {
browsers: [
{
browser: "chromium",
outputName: "chromium",
},
{
browser: "firefox",
outputName: "firefox",
},
],
},
},
mocha: {
reporterOptions: {
"codeceptjs-cli-reporter": {
stdout: "-",
options: {
verbose: true,
debug: true,
steps: true,
},
},
mochawesome: {
stdout: "./output/console.log",
options: {
reportDir: "./output",
reportFilename: "report",
},
},
"mocha-junit-reporter": {
stdout: "./output/console.log",
options: {
mochaFile: "./output/result.xml",
jenkinsMode: true,
attachments: true, //add screenshot for a failed test
},
},
},
},

name: "my",
};
```

Test suites:

SuiteA_test.ts
```ts
Feature("SuiteA");

Scenario("Scenario 1", ({ I }) => {
I.say("Executing Scenario 1 in Suite A");
});

Scenario("Scenario 2", ({ I }) => {
I.say("Executing Scenario 2 in Suite A");
});
```

SuiteB_test.ts
```ts
Feature("SuiteB");

Scenario("Scenario 1", ({ I }) => {
I.say("Executing Scenario 1 in Suite B");
});

Scenario("Scenario 2", ({ I }) => {
I.say("Executing Scenario 2 in Suite B");
});
```

**Used SW:**

- CodeceptJS 3.7.6
- mochawesome 7.1.4
- mocha-junit-reporter 2.2.1
- mocha-multi 1.1.7

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.