Too many duplicates in hsc command line arguments, causing build failures due to GCC argument limits
- Dominant language
- Haskell
- Stars
- 1.7k
- Forks
- 750
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 28
Description
I couldn't find an open bug report for this. Feel free to close this as a duplicate if it already exists!
**Describe the bug**
When `cabal` invokes `hsc`, it builds a list of arguments that among other things, include linker arguments for linking against all of the package's (transitive) dependencies' C dependencies:
https://github.com/haskell/cabal/blob/e7657ad319b9a5b50718262488d17aad915aad90/Cabal/src/Distribution/Simple/PreProcess.hs#L609-L622
Because a lot of packages tend to depend on the same set of libraries, this ends up adding many duplicates of the same argument. Normally this doesn't cause any real problems because `gcc` and the linker will ignore these duplicates, but GCC also has an argument length limit. If you have enough dependencies then you will eventually trip this limit (see https://github.com/NixOS/nixpkgs/issues/41340, though this `hsc` issue is a separate issue).
In a codebase I'm working on right now, `hsc` adds hundreds of the same arguments, causing it to trip this limit. This is an xcerpt from the the `gcc` command line, split on spaces, sorted by the number of occurrences:
```
...
5 -L/nix/store/63w485iy0i6xkbdh2cr0sq98dbbj863v-tzdata-2024a-dev/lib
5 -L/nix/store/971g0fka30djkcww2ddpx7q66dr86c1q-zlib-1.3.1/lib
5 -L/nix/store/dq03m14rqjvbz6pw42g555df4khh4bvr-zlib-1.3.1-dev/lib
5 -Wl,-R,/nix/store/63w485iy0i6xkbdh2cr0sq98dbbj863v-tzdata-2024a-dev/lib
5 -Wl,-R,/nix/store/971g0fka30djkcww2ddpx7q66dr86c1q-zlib-1.3.1/lib
5 -Wl,-R,/nix/store/dq03m14rqjvbz6pw42g555df4khh4bvr-zlib-1.3.1-dev/lib
7 -L/nix/store/sk5if9vmsykj16phb87wv7a8pv08wi62-ghc-9.6.5/lib
7 -Wl,-R,/nix/store/sk5if9vmsykj16phb87wv7a8pv08wi62-ghc-9.6.5/lib
269 -L/nix/store/qi2525fzsnmgayxba5gxc2xi3x80g99x-ncurses-6.4.20221231/lib
269 -Wl,-R,/nix/store/qi2525fzsnmgayxba5gxc2xi3x80g99x-ncurses-6.4.20221231/lib
270 -L/nix/store/3bffbi88dmkpr5ib58wsngqd6bfdjzgj-elfutils-0.191/lib
270 -L/nix/store/nj9g42fdsm8l2z43kfcahch3px2q209a-libffi-3.4.6/lib
270 -L/nix/store/vgzmgmppvpb6gqlc56jhcn007cjn7xg9-gmp-with-cxx-6.3.0/lib
270 -Wl,-R,/nix/store/3bffbi88dmkpr5ib58wsngqd6bfdjzgj-elfutils-0.191/lib
270 -Wl,-R,/nix/store/nj9g42fdsm8l2z43kfcahch3px2q209a-libffi-3.4.6/lib
270 -Wl,-R,/nix/store/vgzmgmppvpb6gqlc56jhcn007cjn7xg9-gmp-with-cxx-6.3.0/lib
```
In this case, the `cabal build` command terminates with the following error:
```
Build profile: -w ghc-9.6.5 -O1
In order, the following will be built (use -v for more details):
- libname-0 (lib) (first run)
Preprocessing library for libname-0..
linking /repo/dist-newstyle/build/x86_64-linux/ghc-9.6.5/libname-0/build/libname/Constant/Buffers_hsc_make.o failed (exit code 1)
rsp file was: "/repo/dist-newstyle/build/x86_64-linux/ghc-9.6.5/libname-0/build/libname/Constant/hsc2hscall2431973-2.rsp"
command was: /nix/store/mpm3i0sbqc9svfch6a17179fs64dz2kv-gcc-wrapper-13.3.0/bin/cc <260 KiB worth of command line arguments>
error: gcc: fatal error: cannot execute ‘/nix/store/zc0nsv23pakbafngjy32kvhfzb16as43-gcc-13.3.0/libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/collect2’: execv: Argument list too long
compilation terminated.
Error: .cabal-wrapped_: Failed to build libname-0.
```
I think simply deduplicating these arguments should fix the issue.
**To Reproduce**
I sadly cannot provide a reproducer because the project in question is not public, but this will happen on any project with a sufficiently large amount of dependencies. The problem is exacerbated by using Nix, since the paths to the library directories end up being much longer that way.
**Expected behavior**
`c2hs`/`hsc` should compile the module as expected.
**System information**
- Ubuntu 22.04
- nixpkgs pin `1e3deb3d8a86a870d925760db1a5adecc64d329d`
- GHC 9.6.5 provided through Nix
- Cabal 3.10.3.0 provided through Nix
**Additional information**
I can provide a patch for this if needed. There may also be other command line invocations with similar problems.
Contributor guide
Research direction
Start in Cabal/src/Distribution/Simple/PreProcess.hs around lines 609-622, where cabal assembles the hsc linker arguments. Reproduce the hsc build with a sufficiently dependency-heavy project if available and verify that repeated linker arguments no longer cause GCC's argument-list limit failure; the issue links a merged pull request for this work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- build-system, cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100