Mochawesome/Testomatio/junit reporter: run-workers: Running of test with --suite doesn't aggregate results from all suites per browser

Inizia da codecept.conf.ts e riproduci il problema usando codeceptjs run-workers 3 dev --by suite --reporter mocha-multi con SuiteA_test.ts e SuiteB_test.ts. Traccia il percorso dei risultati dei worker fino ai percorsi di output configurati di Mochawesome e mocha-junit-reporter; il lavoro è completato quando gli artefatti console.log, HTML, JSON e XML sia di chromium sia di firefox contengono entrambe le suite.

The issue is even in older versions, e.g. in 3.5.3. Retested with the latest 3.7.6 and the issue persists.

• npm i mochawesome
• npm i mocha-junit-reporter
• npm i mocha-multi
• Run the tests codeceptjs run-workers 3 dev --by suite --reporter mocha-multi (parallelization per browser and per suite). Another variant is --suites instead of --by suite, but it behaves the same way.

$ codeceptjs run-workers 3 dev --by suite --reporter mocha-multi
CodeceptJS v3.7.6 #StandWithUkraine
Running tests in 3 workers...

[Worker 03]   ✔ SuiteB › Scenario 1 in 4ms
[Worker 03]   ✔ SuiteB › Scenario 2 in 1ms
[Worker 01]   ✔ SuiteA › Scenario 1 in 5ms
[Worker 01]   ✔ SuiteA › Scenario 2 in 1ms
[Worker 02]   ✔ SuiteA › Scenario 1 in 3ms
[Worker 02]   ✔ SuiteA › Scenario 2 in 1ms
[Worker 04]   ✔ SuiteB › Scenario 1 in 4ms
[Worker 04]   ✔ SuiteB › Scenario 2 in 1ms

  OK  | 8 passed   // 3s

Expected result:

• output/console.log contains data from both test suites
• output/dev_chromium1/report.html contains data from both test suites
• output/dev_chromium1/report.json contains data from both test suites
• output/dev_chromium1/result.xml contains data from both test suites
• output/dev_firefox1/report.html contains data from both test suites
• output/dev_firefox1/report.json contains data from both test suites
• output/dev_firefox1/result.xml contains data from both test suites

This way it works with run-multiple (parallelization per browser, but test suites are run in sequence) ✔️

Actual result:

• All the mentioned files contains data from the SuiteA only. The SuiteB is missing there 🐞 . A problem seems to be that test suites overwrite each other's results when running in threads.

Config codecept.conf.ts:

import { setHeadlessWhen, setCommonPlugins } from "@codeceptjs/configure";
// turn on headless mode when running with HEADLESS=true environment variable
// export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);

// enable all common plugins https://github.com/codeceptjs/configure#setcommonplugins
setCommonPlugins();

export const config: CodeceptJS.MainConfig = {
  tests: "./*_test.ts",
  output: "./output",
  helpers: {
    Playwright: {
      browser: "chromium",
      url: "http://localhost",
      show: true,
    },
    Mochawesome: {
      uniqueScreenshotNames: true,
    },
  },
  include: {
    I: "./steps_file",
  },
  multiple: {
    dev: {
      browsers: [
        {
          browser: "chromium",
          outputName: "chromium",
        },
        {
          browser: "firefox",
          outputName: "firefox",
        },
      ],
    },
  },
  mocha: {
    reporterOptions: {
      "codeceptjs-cli-reporter": {
        stdout: "-",
        options: {
          verbose: true,
          debug: true,
          steps: true,
        },
      },
      mochawesome: {
        stdout: "./output/console.log",
        options: {
          reportDir: "./output",
          reportFilename: "report",
        },
      },
      "mocha-junit-reporter": {
        stdout: "./output/console.log",
        options: {
          mochaFile: "./output/result.xml",
          jenkinsMode: true,
          attachments: true, //add screenshot for a failed test
        },
      },
    },
  },

  name: "my",
};

Test suites:

SuiteA_test.ts

Feature("SuiteA");

Scenario("Scenario 1", ({ I }) => {
  I.say("Executing Scenario 1 in Suite A");
});

Scenario("Scenario 2", ({ I }) => {
  I.say("Executing Scenario 2 in Suite A");
});

SuiteB_test.ts

Feature("SuiteB");

Scenario("Scenario 1", ({ I }) => {
  I.say("Executing Scenario 1 in Suite B");
});

Scenario("Scenario 2", ({ I }) => {
  I.say("Executing Scenario 2 in Suite B");
});

Used SW:

• CodeceptJS 3.7.6
• mochawesome 7.1.4
• mocha-junit-reporter 2.2.1
• mocha-multi 1.1.7