v2-build and v2-haddock succed with either `--enable-tests` or `--enable-benchmarks`, but not with both

**Describe the bug**
I have two libraries (`pomaps` used by `datafix`) that try to minimize its dependency footprint by offering a flag that will include the necessary definitions from `lattices` instead of depending on it directly (I say this because I think that's relevant to the problem). I can run tests and benchmarks just fine, but [it breaks in haskell-ci](https://travis-ci.org/sgraf812/datafix/jobs/635190513). Ultimately, I could reproduce it locally.

**To Reproduce**
Steps to reproduce the behavior:
```bash
$ git clone https://github.com/sgraf812/datafix
$ cd datafix/
$ git checkout 4d101a402f9936ba018fe96483701bb74a95b724
$ cabal v2-build --enable-tests
... just fine ...
$ cabal v2-build --enable-benchmarks
... just fine ...
$ cabal v2-build --enable-tests --enable-benchmarks
...
src/Datafix/MonoMap.hs:17:1: error:
Could not load module ‘Algebra.PartialOrd’
It is a member of the hidden package ‘lattices-2.0.1’.
Perhaps you need to add ‘lattices’ to the build-depends in your .cabal file.
Use -v to see a list of the files searched for.
|
17 | import Algebra.PartialOrd
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
```

Note that apparently it tries to rebuild the library, and doesn't even get as far as building tests or benchmarks.

The same happens with `v2-haddock` instead of `v2-build`.

**Expected behavior**
Builds successfully, no errors, CI turns green

**System information**
- NixOS 19.09
- Cabal 2.4.1.0, GHC 8.6.5