karma-runner / karma-runner/karma

Improve karma to not always run the whole test suite

Open
#1,507 18 comments 67 reactions 0 assignees View on GitHub
discuss
Dominant language
JavaScript
Stars
12k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

Hi there,

my goal is to improve karma to not always run the whole test suite when a file changes.
To avoid lots of effort going in the wrong direction I would like to talk about this first, before starting the implementation.
## Ideal Goal

Improve karma to not always run the whole test suite when a file changes
## Motivation

This is an important issue in larger projects with a lot of tests. Changing a file (using autoWatch or grunt-karma) usually triggers a run of the whole test suite. This is not always necessary and can be avoided.

There are two isolated sets of files that usually trigger a re-run of all the tests:
1. source files
2. spec files

Regarding the first category one cannot do much easily. What one can do, is to derive the set of spec files four each source file from the first test run (maybe Istanbul can help here?). This is not what my proposal is about (although it should be done, but not as a first step).

**What I want to improve is the second set of files: when changing a spec file, karma should run only this particular spec file**

So let's talk about the ...
## ... Realistic Goal

Improve karma to not run the whole test suite when only a **spec** file changes
## Proposed changes (i.e. backwards-compatible additions)
1. Add a sixth configuration option for the files pattern object (current keys are _pattern_, _watched_, _included_, _served_, _nocache_). It could be named `isspec` and defaulting to false. If set to true for file _A_, karma could (when the watcher detects a change in _A_) load (i.e. include as in the _included_ setting) only those files who have `isspec == false` plus _A_. Conversely, if the watcher detects a change in file _B_ which we assume to have `isspec == false`, then it would just load the files as usual.
2. Change No. 1 only works when running karma standalone with **autoWatch** option. It does not work in conjunction with grunt-karma (e.g.). Here, we can have a karma server started in background and tests are only invoked when the build-system detects a change (grunt-karma then uses [the runner api to invoke a test run](https://github.com/karma-runner/grunt-karma/blob/5d1881c951d0e1d2a13e48219226e0c0a3335b80/tasks/grunt-karma.js#L108)) **. The source code reveals: the whole config object is passed to the runner. But the runner [does not respect all of its properties](https://github.com/karma-runner/karma/blob/master/lib/middleware/runner.js#L47-L83). The easiest solution would be to allow for a complete reconfiguration of _config.files_ (just using what is passed to the runner). However, this might affect performance, because if a developer switches between source and spec files (making changes), karma would have to re-do all the globbing again and again. I have the feeling that additional care has to be taken to ensure that this performance loss does not outweigh the benefits of switching between different _config.files_ configurations.

*\* The ideal setup I would like to support involves using (1) grunt-watch, setting up two different watch-targets, one for watching the source files and one for watching the spec files, and using (2) grunt-newer for the spec-files watch task to be able to dynamically pass only the changed files to the _grunt-files array_, e.g. like so: 'newer:karma:run'. Respecting the _grunt-files array_ was only "recently" (in January) [made possible](https://github.com/karma-runner/grunt-karma/commit/cb53deaef6da756be55e35c7d9fa57b84afda2ed#diff-a6d84f8e02b7bd2abf8354f465508f43).
## What do you think?

What do you think? If anything is unclear, I'm happy to try and explain more thoroughly.
I'm feeling that especially change No. 2 could be solved differently in many ways (e.g. letting grunt-karma do more of the work, like WebStorm does, [for which the changes in runner.js were introduced in the first place](https://github.com/karma-runner/karma/commit/b598106de1295f3e1e58338a8eca2b60f99175c3), i.e. the changes to allow for notification on added/removed/changed files).

Regards,
Nicolas

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.