Adding to `extra-lib-dirs` in `.cabal/config` rebuilds everything

I added directories to `extra-lib-dirs` and `extra-include-dirs` in my `.cabal/config`, and now any `cabal build` will rebuild all dependencies, regardless whether they are affected by this change or not.

The current situation is sound and thus an improvement over:
- #2662

However, rebuilding everything is a crude solution and can be quite costly (e.g. `pandoc` takes 1h to build from scratch).
Just _adding_ library directories should be _monotonic_ and not require rebuilding everything.