codeceptjs / codeceptjs/CodeceptJS

Running tests with workers causing incorrect failure / passing count

Ouverte
#2,158 11 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
JavaScript
Étoiles
4.2k
Forks
757
Merge moyen
2 j 9 h
PR mergées (30 j)
16

Description

#### What are you trying to achieve?
To run a series of test features expect 0 or positive failures, or 100% success across all 78 tests.

#### What do you get instead?
In the output below, one test failed but the output says all test have passed (although there are 78 total, the count says 77.)
```
[5] ✔ [testname]
[2] ✔ [testname]
[8] ✖ [testname] 264128ms
[8] [featurename] --
[8] ✔ [testname]
[5] ✔ [testname]
[2] ✔ [testname]
[2] ✔ [testname]
[5] ✔ [testname]
[5] ✔ [testname]
[2] ✔ [testname]
[5] ✔ [testname]
[2] ✔ [testname]
[5] ✔ [testname]
[5] ✔ [testname]
[5] ✔ [testname]
[5] ✔ [testname]
[5] ✔ [testname]
[5] ✔ [testname]

OK | 77 passed // 1106.326s
/tmp/build/3e139e9d
+ '[' 0 -ne 0 ']'
```

In other instances, with no pipeline changes, it would say all tests passed with -1 failed.
```
FAIL | 78 passed, -1 failed // 1144.407s
```

Codecept is being run with:
```
codeceptjs run-workers --suites 8 --grep @full-suite
```

These asynchronous tests were passing OK for some time. They are all being ran within a codecept container.

### Details

* CodeceptJS version: v2.4.1
* NodeJS Version: v12.10.0
* Operating System: Not sure
* puppeteer || webdriverio || protractor || testcafe version (if related) Not sure
* Configuration file:

```js
exports.config = {
tests: './tests/*.js',
timeout: 10000,
output: './output',
helpers: {
Puppeteer: {
chrome: { args: ['--no-sandbox'] },
url: 'http://localhost:8000',
show: false,
restart: false,
keepBrowserState: true,
windowSize: '1200x800',
waitForNavigation: 'networkidle0',
waitForTimeout: 30000,
},
DomHelpers: { require: './helpers/DomHelpers.js' },
},
include: {
I: './steps/steps_file.js',
[more stuff]
},
plugins: {
retryFailedStep: {
enabled: true,
retries: 5,
minTimeout: 500,
},
[customplugin]
},
bootstrap: false,
// bail will fail hard after one test failure
mocha: { bail: false },
name: '[name]',
};
```

Hopefully the info I've provided is sufficient -- please let me know if I can provide anything else!

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.