commercialhaskell / commercialhaskell/stack

'stack test --no-run-benchmarks' doesn't build the all benchmarks

Open
#4,045 11 comments 0 reactions 0 assignees View on GitHub
awaiting pull request in progress newcomer friendly type: enhancement work-in-progress
Dominant language
Haskell
Stars
4.1k
Forks
850
Avg merge
10h 37m
Merged PRs (30d)
4

Description

Using stack version 1.7.1 (current latest).

### General summary/comments (optional)

The `--no-run-benckmarks` option to the `test` command does not work as described in the `stack test --help` output.
```
> stack test --help | grep -1 run-benchmarks
[--no-run-tests] [--[no-]bench]
[--ba|--benchmark-arguments BENCH_ARGS] [--no-run-benchmarks]
[--[no-]reconfigure] [--[no-]cabal-verbose]
--
templates from `cabal bench`
--no-run-benchmarks Disable running of benchmarks. (Benchmarks will still
be built.)
```

### Steps to reproduce

This passes:
```
https://github.com/input-output-hk/cardano-sl
cd cardano-sl
git checkout c3a292fee627bc9e76bb5b4896df59471a4dbdc4 -b current-develop
stack test --no-run-benchmarks cardano-sl
```
but then this:
```
stack bench --no-run-benchmarks cardano-sl
```
fails with a build error (in one of the benchmarks) that should have been caught by the `stack test --no-run-benchmarks` command.

### Expected

In this case, since there actually is a build error when building the benchmarks, `stack test --no-run-benchmarks` should exit with a non-zero exit code. This is important because we want changes in libraries that break the benchmarks to be caught in CI.

### Actual

The 'stack test --no-run-benchmarks' command doesn't build the all benchmarks and therefore does not fail the CI build.

### Stack version

```
> stack --version | head
Version 1.7.1, Git revision c4ec53e3807cc7a768cc833a3dd41b9d4db60d72 (dirty) (3 commits) x86_64
Compiled with:
- Cabal-2.2.0.1
- Glob-0.9.2
- HUnit-1.6.0.0
- QuickCheck-2.11.3
- StateVar-1.1.1.0
- aeson-1.2.4.0
- aeson-compat-0.3.7.1
- annotated-wl-pprint-0.7.0
```

The git version above is not the git version of stack, but rather the git version of the repository where I keep metadata for creating a Debian package for my own use. My packaging metadata builds the version specified from the Hackage tarball.

### Method of installation

* Via cabal-install

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.