Only run tests for files that have changed in watch mode

TL;DR: Elm is pure and static, so the scope of changes can be computed statically with relative ease. If the tests pass, then a change is made, only a subset of tests need to be run to determine that all tests would pass if they were to be run. The main benefit is speeding up test runs.

Motivation

This is what the module dependency graph looks like for elm-visualization (some simplifactions and omissions have been made):

This will run a good number of tests. However one can quickly notice that following the graph upwards allows one to only run the tests that could have possibly changed. For example, if I change Log.elm, I can color the graph like this by following the arrows upwards:

From this it's fairly obvious that I would only need to run the tests in AxisTest.elm, as that is the only module that could have been possibly affected by the change that has been made.

(As an aside, this could be done not only at the module level, but at the value level, however I think that would transform this from a straightforward problem to something equivalent to doing dead-code elimination.)

Proposed solution

When running tests in --watch mode, the test runner would maintain an in memory graph of the various module dependencies. Initially that would mean scanning all the source and test files being watched and parsing out the import statements. Given elm's syntax, this shouldn't be too difficult. These then need to be assembled into a DAG.

When a file watch event arrives, the arrows going from the module/file changed will be deleted. That file than can be rescanned and new arrows created. This way the graph can be kept up to date efficiently.

Chances are that this would make a decent separate npm package.

Then the graph is scanned in reverse to the direction of the arrows and any modules that expose suite : Test would be added to a list to run. (Obviously this depends on #107).

To improve usability, tests that failed in a previous run should either still be in the run list, or simply their output should be simply reported again, so that failures are easily visible.

Also the tests that were actually run should be printed to the console, so that it is easy to reproduce the same result, so:

Running 91 tests. To reproduce these results, run: elm-test --seed 23409321 tests/AxisTest.elm

Limitations

• This won't necessarily work for Native/Kernel code, as the strong guarantees that enable this to work reliably for elm code don't necessarily work for those.
• When running in watch mode, the first run will be probably slightly slower on large projects, as the dependency graph would need to be built.

Start at the elm-test --watch entry point and review how watched source and test files are discovered; the issue proposes parsing import statements into an in-memory dependency graph. Done means changed files trigger only affected suites, prior failures remain visible, and the console prints the suites and reproduction command, while accounting for the dependency on #107.