haskell / haskell/cabal

Too many duplicates in hsc command line arguments, causing build failures due to GCC argument limits

Open
#10,158 2 comments 9 reactions 0 assignees View on GitHub
needs triage status: consider closing type: bug
Dominant language
Haskell
Stars
1.7k
Forks
750
Avg merge
4d 3h
Merged PRs (30d)
28

Description

I couldn't find an open bug report for this. Feel free to close this as a duplicate if it already exists!

**Describe the bug**
When `cabal` invokes `hsc`, it builds a list of arguments that among other things, include linker arguments for linking against all of the package's (transitive) dependencies' C dependencies:

https://github.com/haskell/cabal/blob/e7657ad319b9a5b50718262488d17aad915aad90/Cabal/src/Distribution/Simple/PreProcess.hs#L609-L622

Because a lot of packages tend to depend on the same set of libraries, this ends up adding many duplicates of the same argument. Normally this doesn't cause any real problems because `gcc` and the linker will ignore these duplicates, but GCC also has an argument length limit. If you have enough dependencies then you will eventually trip this limit (see https://github.com/NixOS/nixpkgs/issues/41340, though this `hsc` issue is a separate issue).

In a codebase I'm working on right now, `hsc` adds hundreds of the same arguments, causing it to trip this limit. This is an xcerpt from the the `gcc` command line, split on spaces, sorted by the number of occurrences:

```
...
5 -L/nix/store/63w485iy0i6xkbdh2cr0sq98dbbj863v-tzdata-2024a-dev/lib
5 -L/nix/store/971g0fka30djkcww2ddpx7q66dr86c1q-zlib-1.3.1/lib
5 -L/nix/store/dq03m14rqjvbz6pw42g555df4khh4bvr-zlib-1.3.1-dev/lib
5 -Wl,-R,/nix/store/63w485iy0i6xkbdh2cr0sq98dbbj863v-tzdata-2024a-dev/lib
5 -Wl,-R,/nix/store/971g0fka30djkcww2ddpx7q66dr86c1q-zlib-1.3.1/lib
5 -Wl,-R,/nix/store/dq03m14rqjvbz6pw42g555df4khh4bvr-zlib-1.3.1-dev/lib
7 -L/nix/store/sk5if9vmsykj16phb87wv7a8pv08wi62-ghc-9.6.5/lib
7 -Wl,-R,/nix/store/sk5if9vmsykj16phb87wv7a8pv08wi62-ghc-9.6.5/lib
269 -L/nix/store/qi2525fzsnmgayxba5gxc2xi3x80g99x-ncurses-6.4.20221231/lib
269 -Wl,-R,/nix/store/qi2525fzsnmgayxba5gxc2xi3x80g99x-ncurses-6.4.20221231/lib
270 -L/nix/store/3bffbi88dmkpr5ib58wsngqd6bfdjzgj-elfutils-0.191/lib
270 -L/nix/store/nj9g42fdsm8l2z43kfcahch3px2q209a-libffi-3.4.6/lib
270 -L/nix/store/vgzmgmppvpb6gqlc56jhcn007cjn7xg9-gmp-with-cxx-6.3.0/lib
270 -Wl,-R,/nix/store/3bffbi88dmkpr5ib58wsngqd6bfdjzgj-elfutils-0.191/lib
270 -Wl,-R,/nix/store/nj9g42fdsm8l2z43kfcahch3px2q209a-libffi-3.4.6/lib
270 -Wl,-R,/nix/store/vgzmgmppvpb6gqlc56jhcn007cjn7xg9-gmp-with-cxx-6.3.0/lib
```

In this case, the `cabal build` command terminates with the following error:

```
Build profile: -w ghc-9.6.5 -O1
In order, the following will be built (use -v for more details):
- libname-0 (lib) (first run)
Preprocessing library for libname-0..
linking /repo/dist-newstyle/build/x86_64-linux/ghc-9.6.5/libname-0/build/libname/Constant/Buffers_hsc_make.o failed (exit code 1)
rsp file was: "/repo/dist-newstyle/build/x86_64-linux/ghc-9.6.5/libname-0/build/libname/Constant/hsc2hscall2431973-2.rsp"
command was: /nix/store/mpm3i0sbqc9svfch6a17179fs64dz2kv-gcc-wrapper-13.3.0/bin/cc <260 KiB worth of command line arguments>
error: gcc: fatal error: cannot execute ‘/nix/store/zc0nsv23pakbafngjy32kvhfzb16as43-gcc-13.3.0/libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/collect2’: execv: Argument list too long
compilation terminated.

Error: .cabal-wrapped_: Failed to build libname-0.
```

I think simply deduplicating these arguments should fix the issue.

**To Reproduce**
I sadly cannot provide a reproducer because the project in question is not public, but this will happen on any project with a sufficiently large amount of dependencies. The problem is exacerbated by using Nix, since the paths to the library directories end up being much longer that way.

**Expected behavior**
`c2hs`/`hsc` should compile the module as expected.

**System information**
- Ubuntu 22.04
- nixpkgs pin `1e3deb3d8a86a870d925760db1a5adecc64d329d`
- GHC 9.6.5 provided through Nix
- Cabal 3.10.3.0 provided through Nix

**Additional information**
I can provide a patch for this if needed. There may also be other command line invocations with similar problems.

Contributor guide

Open the contributing guide

Research direction

Start in Cabal/src/Distribution/Simple/PreProcess.hs around lines 609-622, where cabal assembles the hsc linker arguments. Reproduce the hsc build with a sufficiently dependency-heavy project if available and verify that repeated linker arguments no longer cause GCC's argument-list limit failure; the issue links a merged pull request for this work.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
build-system, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.