--enable-tests / --test-option(s) causes recalculation

**Describe the bug**

Repeatedly swapping between

```
cabal v2-build
```

and

```
cabal v2-build --enable-tests
```

causes a recalculation of all the dependencies (but no recompilation, after the first full build).

**To Reproduce**

See above

**Expected behavior**

I would expect (after the first full compilation) no further work by cabal.

**System informataion**

```
cabal --version
cabal-install version 2.4.1.0
compiled using version 2.4.1.0 of the Cabal library
```

on any GNU/Linux with any ghc.

**Additional context**

My projects are quite large with multiple `.cabal` packages and many hackage dependencies. Hence recalculation is quite costly (e.g. 30 seconds every time the dependencies are recomputed). I'm also using a freeze file.

In addition,

```
cabal v2-run tasty
```

vs

```
cabal v2-run tasty --enable-tests
```

exhibits the same problem.

A workaround is to always use `--enable-tests` even when doing `v2-run tasty` and even though it is redundant.

I also typically use `-O0` for these commands, but that's probably irrelevant (much faster dev cycle).

Reproduce the issue by alternating `cabal v2-build` with `cabal v2-build --enable-tests`, and similarly compare the two `cabal v2-run tasty` commands. Start by tracing how `--enable-tests` and `--test-option(s)` affect dependency recalculation, including projects using a freeze file. Done means repeated command changes avoid unnecessary dependency recalculation after the initial build.

Written by the indexing model from the issue text.