haskell / haskell/cabal

Cabal's GHC batch compilation uses too much memory when the module count is high

Open
#6,658 5 comments 3 reactions 0 assignees View on GitHub
type: enhancement type: performance
Dominant language
Haskell
Stars
1.7k
Forks
750
Avg merge
4d 3h
Merged PRs (30d)
28

Description

**Describe the bug**
Cabal uses GHC in batch mode (--make) when compiling libraries and executables.
This works for small projects with small number of modules, and it also prevents to reload .hi files multiple times. However for large project like pandoc which have 186 modules, GHC consumes more and more memory during the compilation process. In the end it uses 6 GB memory. This is wasteful, the GC just copies unused in-memory interface file data.
I believe in case of large packages with many modules it would be more efficient to compile modules separately (ghc -c). Hadrian does exactly this. It first calculates the dependencies of modules and saves it to file (as cache) then compiles each module traversing the dependency tree in topological order. I think Cabal should do the same, especially for large projects.

**To Reproduce**
Steps to reproduce the behavior:

```
$ cabal v2-build pandoc
```

Please use version-prefixed commands (e.g. `v2-build` or `v1-build`) to avoid ambiguity.

**Expected behavior**
When compiling pandoc app with it 186 (approx) modules, a single ghc process will use more then 6GB memory, the module compilation process slows down gradually.

**System information**
- Operating system: Ubuntu 16.04
- `cabal`, `ghc` versions: cabal 3.3, ghc 8.12

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported case with `cabal v2-build pandoc` and record memory use during compilation. Read the batch compilation implementation and compare its behavior with Hadrian's dependency calculation, cache, and topological compilation approach. Done means large packages such as pandoc compile without the reported single-process memory growth and slowdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
build-system, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.