Stack rebuilds whole application in file-watch mode, when it should be able to incrementally recompile

I have opened this issue from https://github.com/yesodweb/yesod/issues/1661, where the context of my issue is I wish to switch between `yesod-devel` for my yesod application, and run `stack test`, without needing to re-compile the whole of my app. Doing so currently seems to trigger a full re-build, and I have narrowed down my problem to the stack commands I believe are being executed in this context.

If I build my application with tests like so:

```
stack build --fast --test --flag myapp:library-only --flag myapp:dev

-- outputs:
myapp> build (lib + exe + test)
```

And then run this command (which is the raw command from yesod devel, minus the `--exec "/home/dan/.local/bin/yesod devel-signal"` flag):

```
stack build --fast --file-watch myapp:lib --flag myapp:dev --flag myapp:library-only
```

I get this output:

```
Success! Waiting for next file change.
Type help for available commands. Press enter to force a rebuild.
```

Then when I make a file change, this results in the whole `lib` project needing to be rebuilt, and the following output:

```
myapp> build (lib)
```

My thought process here though is that the whole library has been built as part of the first test command, and therefore should not need to trigger an entire re-build from scratch, and instead should be able to incrementally re-compile from the file I have changed and onwards.

---

As part of my troubleshooting with stack I have been adding logging here and there to some of the files in `src/Stack/Build`, and have shifted a few things around in order to make the code a bit easier to follow, and also get at certain values in different places.

Would a refactor PR be welcomed with these changes to the project?