commercialhaskell / commercialhaskell/stack
'stack test --no-run-benchmarks' doesn't build the all benchmarks
- Dominant language
- Haskell
- Stars
- 4.1k
- Forks
- 850
- Avg merge
- 10h 37m
- Merged PRs (30d)
- 4
Description
Using stack version 1.7.1 (current latest).
### General summary/comments (optional)
The `--no-run-benckmarks` option to the `test` command does not work as described in the `stack test --help` output.
```
> stack test --help | grep -1 run-benchmarks
[--no-run-tests] [--[no-]bench]
[--ba|--benchmark-arguments BENCH_ARGS] [--no-run-benchmarks]
[--[no-]reconfigure] [--[no-]cabal-verbose]
--
templates from `cabal bench`
--no-run-benchmarks Disable running of benchmarks. (Benchmarks will still
be built.)
```
### Steps to reproduce
This passes:
```
https://github.com/input-output-hk/cardano-sl
cd cardano-sl
git checkout c3a292fee627bc9e76bb5b4896df59471a4dbdc4 -b current-develop
stack test --no-run-benchmarks cardano-sl
```
but then this:
```
stack bench --no-run-benchmarks cardano-sl
```
fails with a build error (in one of the benchmarks) that should have been caught by the `stack test --no-run-benchmarks` command.
### Expected
In this case, since there actually is a build error when building the benchmarks, `stack test --no-run-benchmarks` should exit with a non-zero exit code. This is important because we want changes in libraries that break the benchmarks to be caught in CI.
### Actual
The 'stack test --no-run-benchmarks' command doesn't build the all benchmarks and therefore does not fail the CI build.
### Stack version
```
> stack --version | head
Version 1.7.1, Git revision c4ec53e3807cc7a768cc833a3dd41b9d4db60d72 (dirty) (3 commits) x86_64
Compiled with:
- Cabal-2.2.0.1
- Glob-0.9.2
- HUnit-1.6.0.0
- QuickCheck-2.11.3
- StateVar-1.1.1.0
- aeson-1.2.4.0
- aeson-compat-0.3.7.1
- annotated-wl-pprint-0.7.0
```
The git version above is not the git version of stack, but rather the git version of the repository where I keep metadata for creating a Debian package for my own use. My packaging metadata builds the version specified from the Hackage tarball.
### Method of installation
* Via cabal-install
Contributor guide
Assessment
This issue has not been assessed yet.