karma-runner / karma-runner/karma

Option to fail on skipped tests

Open
#3,284 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
12k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

This is a feature request to add ability to fail (exit with non-0 code) on skipped tests. This is to address an issue of someone accidentally using focused Jasmine tests (fdescribe/fit) and checking this into code repository.

The run then exits with:

```
PhantomJS 2.1.1 (Windows 8.0.0): Executed 1 of 83 (skipped 82) SUCCESS (0.13 secs / 0.059 secs)
TOTAL: 1 SUCCESS
```

Printing exit code:
```
$ echo $?
0

```

This is needed to fail such a run in Maven which uses plugins that rely on exit code to know whether to fail the 'test' goal or not. There is no other known reliable way of providing exit status from Karma. The plugins cannot read output reports as these could be wildly different depending on which report was printed.

Line 72 in frontend-maven-plugin by eirslett:
https://github.com/eirslett/frontend-maven-plugin/blob/master/frontend-plugin-core/src/main/java/com/github/eirslett/maven/plugins/frontend/lib/ProcessExecutor.java

Line 277 in maven-karma-plugin by kelveden:
https://github.com/karma-runner/maven-karma-plugin/blob/master/src/main/java/com/kelveden/karma/StartMojo.java

The feature would be similar to the option to 'failOnEmptyTestSuite' but called something like 'failOnSkippedTests'

### Environment Details

- Karma version (output of `karma --version`):
Karma version: 3.1.4

"karma-jasmine": "^2.0.1",
"karma-phantomjs-launcher": "^1.0.4",
"karma-htmlfile-reporter": "^0.3.7",
"karma-jasmine-html-reporter": "^1.4.0",
"karma-spec-reporter": "0.0.32",

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing failOnEmptyTestSuite option and its tests, then trace how Karma determines its process exit code after skipped tests. Add the analogous configuration behavior and verify that a run with skipped tests exits non-zero while normal runs retain their current status.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.