commercialhaskell / commercialhaskell/stack

How to generate executable coverage reports with stack

Open
#3,307 6 comments 0 reactions 0 assignees View on GitHub
component: coverage type: discuss
Dominant language
Haskell
Stars
4.1k
Forks
850
Avg merge
10h 37m
Merged PRs (30d)
4

Description

So my code is arranged as four executables (i.e. most/all of the code is in "module Main"/main), with an insignificantly small bit of shared library. I would very much like to see coverage information on them, and this turns out to be Hard (tm). It is worth noting that if you want to do this *without* fighting GHC and stack every step of the way, the solution is probably to make your executables be teeny tiny stubs, and move everything into library modules, and I'm probably going to try that next, but having made this work I thought it was worth sharing, and some of it might be appropriate to roll into stack.

I have managed to hack together a script that does this, in somewhat grotesque fashion, but everything in here is stuff that stack probably *could* do without too much trouble? The ghc options in particular.

https://github.com/rlpowell/hblog/commit/b3fb0173cbb4d2a4b1fcdee66019baee5595c769

And you can see the output at:

http://users.teddyb.org/~rlpowell/media/public/hblog-coverage-2017-08-02/

The key bits here:

1. Every executable gets its own .mix directory; this is accomplished by having bits like:

ghc-options: -threaded -Wall -fhpc -hpcdir hpc/tiki_to_md/

This is super important because from GHC's POV, all of these are running the same module, and they all generate Main.mix (and there doesn't seem to be any way to have a program's main module be named anything *but* Main), so they all collide horribly unless so separated.

2. I have to add a space to the end of each file to make sure they recompile with said coverage options -_-; see #3306

3. Because my test suite runs the various executables in various places and a various number of times, I have to collate them using "stack exec hpc sum", and in the case of the test suite itself I have to rename its tix file from hblog-test.tix to test.tix (I have no idea why).

4. The actual coverage graphs are then generate as several calls like:

stack exec hpc -- markup --hpcdir=hpc/$name/ --destdir=cov/$name/ --verbosity=2 hpc/$name.tix

5. There are several bits here that are specific to my environment. In particular, ignore all the rsyncs, chmods, and zipping/unzipping.

Contributor guide

Open the contributing guide

Research direction

Start with the linked hblog commit and review how its ghc-options, per-executable hpc directories, stack exec hpc sum, and hpc markup commands work together. Check the related #3306 issue for the recompilation problem. Done means Stack can support the described executable coverage workflow without the manual script steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
build-system, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.