karma-runner / karma-runner/karma-qunit
QUnit.todo() with a failure are reported as failling
- Dominant language
- JavaScript
- Stars
- 52
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
I had the use case of a test being properly implemented but the feature code being incomplete and failing. When running the tests with the QUnit web interface, the test is not erroring out and is properly labeled `todo`.

However when running karma, the test is wrongfully reported as failing:
```
SUMMARY:
✖ 1 test failed
FAILED TESTS:
ext.graph.visualEditor
✖ ve.ui.TableWidget
Chrome 66.0.3359 (Linux 0.0.0)
Died on test #4 at
....
TypeError: Cannot read property '0' of undefined
at VeUiTableWidget.ve.ui.TableWidget.onInsertColumn // expected failure
...
at runTest (node_modules/qunit/qunit/qunit.js:1530:30)
at Test.run (node_modules/qunit/qunit/qunit.js:1516:6)
at node_modules/qunit/qunit/qunit.js:1728:12
at advanceTaskQueue (node_modules/qunit/qunit/qunit.js:1129:6)
at advance (node_modules/qunit/qunit/qunit.js:1110:4)
```
In `lib/adapter.js` I adjusted `runner.testDone` to print the `test` and `testResult` to the console which yields:
```
LOG: Object{
name: 've.ui.TableWidget',
module: 'ext.graph.visualEditor',
skipped: false,
todo: true,
failed: 1,
passed: 3,
total: 4,
LOG: Object{success: false, errors: ['Died on test #4 ...
```
The testResult is set by runner.log which mark it with `success = false`. However in runner.testDone, the `todo` flag should be used to invert the result. Potentially:
```
--- orig.js 2018-06-26 22:31:26.880087441 +0200
+++ new.js 2018-06-26 22:32:36.218186993 +0200
@@ -8,6 +8,11 @@
time: new Date().getTime() - timer
}
+ // For QUnit.todo(): flip success report
+ if ( test.todo === true ) {
+ result.success = !testResult.success
+ }
+
if (result.description.indexOf('global failure') !== -1) {
return
}
```
**Environment**:
* karma 2.0.2
* karma-qunit 2.0.1
* qunit 2.6.0
Contributor guide
Research direction
Start in lib/adapter.js at runner.testDone and trace how runner.log supplies testResult.success for a QUnit.todo() test. Reproduce the provided karma-qunit and QUnit case, then verify that an expected todo failure is reported as todo rather than as an ordinary failing test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100