Turn cabal-install into a build graph orchestrator
- Dominant language
- Haskell
- Stars
- 1.7k
- Forks
- 750
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 28
Description
#976 and #4174 both discuss the idea of `cabal-install` keeping track of a per-module build graph. However, recent architectural changes in `cabal-install` significantly change the design constraints.
Firstly, we now have [`-jsem`](https://github.com/ghc-proposals/ghc-proposals/pull/540). This allows us to recover module-level compilation parallelism when invoking `cabal-install` with `--semaphore`. (We should move towards making it the default.)
The remaining enhancements from #976 are:
1. Cross-package pipelining. If package `q` depends on `p`, we should be able to start compiling modules of `q` once all the modules in `p` on which it transitively depends have been compiled, instead of waiting for the entire of `p` to have been built.
2. Improved recompilation checking. We can avoid the overhead of `ghc --make` invocations (see #6658) and can rebuild at a finer granularity.
Secondly, [HF RFC #060](https://github.com/haskellfoundation/tech-proposals/pull/60) and its implementation in c5f99331266a88a833a69b5766b69aaf68f24097 + 6867dd5da57194337bab0079bbe2f1c21e11662f provides `cabal-install` with a complete picture of what it is building (instead of opaque `./Setup build` invocations which it can't peek into). This allows us to rethink the level of granularity that `cabal-install` is operating at: it can operate on the module level instead of the package level (with some opaque single `./Setup build` nodes for legacy `build-type: Custom` packages).
## New design
Let's look at what `Distribution.Simple.Build.build_setupHooks` currently does:
1. `pre-build`: generate autogenerated files (e.g. paths module), run pre-build rules.
2. Run pre-processors (alex, happy,...).
3. Build all Haskell modules in one go with one `ghc --make` invocation.
4. Build non-Haskell code (e.g. C files).
5. Link.
6. Install/register.
To `cabal-install`, this is one opaque blob today (see `Distribution.Client.ProjectBuilding.rebuildTargets`).
This ticket concretely suggests to make `Cabal` return the individual steps above as separate rules, and implement a graph-based orchestrator in `cabal-install` that executes rule graphs.
For the time being, we would continue to use `ghc --make` to build all Haskell modules in one go. Therefore this ticket is only the first phase in implementing per-module build graphs in `cabal-install`.
The gains we get from this first phase are minor but not insignificant:
- Finer-grained recompilation checking (e.g. can entirely avoid invoking GHC for building Haskell modules if a C file changes).
- Slightly better cross-package pipelining (e.g. we can start running preprocessors in `q` before Haskell module dependencies of `p` have been built).
This ticket is agnostic about the second phase, which concerns splitting up `ghc --make` invocations. We can use many individual one-shot `ghc -c` invocations, but this means a lot more work dealing with interface files. Or we could be more ambitious and use a persistent GHC worker where everything is kept in-memory.
Contributor guide
Research direction
Start by reading Distribution.Simple.Build.build_setupHooks and Distribution.Client.ProjectBuilding.rebuildTargets, then review the linked RFC and implementation commits to understand the build information now available. The first phase is done when Cabal exposes the listed build steps as separate rules and cabal-install can execute those rules as a graph while still using ghc --make for Haskell modules.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100