haskell / haskell/cabal

Turn cabal-install into a build graph orchestrator

Open
#12,121 1 comment 7 reactions 0 assignees View on GitHub
cabal-install: cmd/build type: discussion type: enhancement type: performance
Dominant language
Haskell
Stars
1.7k
Forks
750
Avg merge
4d 3h
Merged PRs (30d)
28

Description

#976 and #4174 both discuss the idea of `cabal-install` keeping track of a per-module build graph. However, recent architectural changes in `cabal-install` significantly change the design constraints.

Firstly, we now have [`-jsem`](https://github.com/ghc-proposals/ghc-proposals/pull/540). This allows us to recover module-level compilation parallelism when invoking `cabal-install` with `--semaphore`. (We should move towards making it the default.)

The remaining enhancements from #976 are:

1. Cross-package pipelining. If package `q` depends on `p`, we should be able to start compiling modules of `q` once all the modules in `p` on which it transitively depends have been compiled, instead of waiting for the entire of `p` to have been built.
2. Improved recompilation checking. We can avoid the overhead of `ghc --make` invocations (see #6658) and can rebuild at a finer granularity.

Secondly, [HF RFC #060](https://github.com/haskellfoundation/tech-proposals/pull/60) and its implementation in c5f99331266a88a833a69b5766b69aaf68f24097 + 6867dd5da57194337bab0079bbe2f1c21e11662f provides `cabal-install` with a complete picture of what it is building (instead of opaque `./Setup build` invocations which it can't peek into). This allows us to rethink the level of granularity that `cabal-install` is operating at: it can operate on the module level instead of the package level (with some opaque single `./Setup build` nodes for legacy `build-type: Custom` packages).

## New design

Let's look at what `Distribution.Simple.Build.build_setupHooks` currently does:

1. `pre-build`: generate autogenerated files (e.g. paths module), run pre-build rules.
2. Run pre-processors (alex, happy,...).
3. Build all Haskell modules in one go with one `ghc --make` invocation.
4. Build non-Haskell code (e.g. C files).
5. Link.
6. Install/register.

To `cabal-install`, this is one opaque blob today (see `Distribution.Client.ProjectBuilding.rebuildTargets`).

This ticket concretely suggests to make `Cabal` return the individual steps above as separate rules, and implement a graph-based orchestrator in `cabal-install` that executes rule graphs.

For the time being, we would continue to use `ghc --make` to build all Haskell modules in one go. Therefore this ticket is only the first phase in implementing per-module build graphs in `cabal-install`.

The gains we get from this first phase are minor but not insignificant:

- Finer-grained recompilation checking (e.g. can entirely avoid invoking GHC for building Haskell modules if a C file changes).
- Slightly better cross-package pipelining (e.g. we can start running preprocessors in `q` before Haskell module dependencies of `p` have been built).

This ticket is agnostic about the second phase, which concerns splitting up `ghc --make` invocations. We can use many individual one-shot `ghc -c` invocations, but this means a lot more work dealing with interface files. Or we could be more ambitious and use a persistent GHC worker where everything is kept in-memory.

Contributor guide

Open the contributing guide

Research direction

Start by reading Distribution.Simple.Build.build_setupHooks and Distribution.Client.ProjectBuilding.rebuildTargets, then review the linked RFC and implementation commits to understand the build information now available. The first phase is done when Cabal exposes the listed build steps as separate rules and cabal-install can execute those rules as a graph while still using ghc --make for Haskell modules.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.