karma-runner / karma-runner/karma
Option to fail on skipped tests
- Dominant language
- JavaScript
- Stars
- 12k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
This is a feature request to add ability to fail (exit with non-0 code) on skipped tests. This is to address an issue of someone accidentally using focused Jasmine tests (fdescribe/fit) and checking this into code repository.
The run then exits with:
```
PhantomJS 2.1.1 (Windows 8.0.0): Executed 1 of 83 (skipped 82) SUCCESS (0.13 secs / 0.059 secs)
TOTAL: 1 SUCCESS
```
Printing exit code:
```
$ echo $?
0
```
This is needed to fail such a run in Maven which uses plugins that rely on exit code to know whether to fail the 'test' goal or not. There is no other known reliable way of providing exit status from Karma. The plugins cannot read output reports as these could be wildly different depending on which report was printed.
Line 72 in frontend-maven-plugin by eirslett:
https://github.com/eirslett/frontend-maven-plugin/blob/master/frontend-plugin-core/src/main/java/com/github/eirslett/maven/plugins/frontend/lib/ProcessExecutor.java
Line 277 in maven-karma-plugin by kelveden:
https://github.com/karma-runner/maven-karma-plugin/blob/master/src/main/java/com/kelveden/karma/StartMojo.java
The feature would be similar to the option to 'failOnEmptyTestSuite' but called something like 'failOnSkippedTests'
### Environment Details
- Karma version (output of `karma --version`):
Karma version: 3.1.4
"karma-jasmine": "^2.0.1",
"karma-phantomjs-launcher": "^1.0.4",
"karma-htmlfile-reporter": "^0.3.7",
"karma-jasmine-html-reporter": "^1.4.0",
"karma-spec-reporter": "0.0.32",
Contributor guide
Research direction
Start by locating the existing failOnEmptyTestSuite option and its tests, then trace how Karma determines its process exit code after skipped tests. Add the analogous configuration behavior and verify that a run with skipped tests exits non-zero while normal runs retain their current status.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100