karma-runner / karma-runner/karma-qunit

QUnit.todo() with a failure are reported as failling

Open
#111 1 comment 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
JavaScript
Stars
52
Forks
37
PR merge metrics
No merged PRs in 30d

Description

I had the use case of a test being properly implemented but the feature code being incomplete and failing. When running the tests with the QUnit web interface, the test is not erroring out and is properly labeled `todo`.

![qunit_todo_is_ok_on_web](https://user-images.githubusercontent.com/281689/41936763-88a254ca-798e-11e8-8e04-fb26fac667c5.png)

However when running karma, the test is wrongfully reported as failing:
```
SUMMARY:
✖ 1 test failed

FAILED TESTS:
ext.graph.visualEditor
✖ ve.ui.TableWidget
Chrome 66.0.3359 (Linux 0.0.0)
Died on test #4 at
....
TypeError: Cannot read property '0' of undefined
at VeUiTableWidget.ve.ui.TableWidget.onInsertColumn // expected failure
...
at runTest (node_modules/qunit/qunit/qunit.js:1530:30)
at Test.run (node_modules/qunit/qunit/qunit.js:1516:6)
at node_modules/qunit/qunit/qunit.js:1728:12
at advanceTaskQueue (node_modules/qunit/qunit/qunit.js:1129:6)
at advance (node_modules/qunit/qunit/qunit.js:1110:4)
```

In `lib/adapter.js` I adjusted `runner.testDone` to print the `test` and `testResult` to the console which yields:
```
LOG: Object{
name: 've.ui.TableWidget',
module: 'ext.graph.visualEditor',
skipped: false,
todo: true,
failed: 1,
passed: 3,
total: 4,
LOG: Object{success: false, errors: ['Died on test #4 ...
```

The testResult is set by runner.log which mark it with `success = false`. However in runner.testDone, the `todo` flag should be used to invert the result. Potentially:

```
--- orig.js 2018-06-26 22:31:26.880087441 +0200
+++ new.js 2018-06-26 22:32:36.218186993 +0200
@@ -8,6 +8,11 @@
time: new Date().getTime() - timer
}

+ // For QUnit.todo(): flip success report
+ if ( test.todo === true ) {
+ result.success = !testResult.success
+ }
+
if (result.description.indexOf('global failure') !== -1) {
return
}
```

**Environment**:
* karma 2.0.2
* karma-qunit 2.0.1
* qunit 2.6.0

Contributor guide

Open the contributing guide

Research direction

Start in lib/adapter.js at runner.testDone and trace how runner.log supplies testResult.success for a QUnit.todo() test. Reproduce the provided karma-qunit and QUnit case, then verify that an expected todo failure is reported as todo rather than as an ordinary failing test.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.