Improved Concurrency when building workspace and packages
Nobody has claimed this yet.
- Dominant language
- OCaml
- Stars
- 1.9k
- Forks
- 500
- Avg merge
- 15h 21m
- Merged PRs (30d)
- 277
Description
When using dune to build packages, the build plan must proceed as follows:
- Build all packages defined in dune.lock
- Build everything defined in the local workspace
These two steps must be done in sequence because to build something in the local workspace, dune must resolve the libraries being used. To resolve any particular library, dune needs to build all packages. This is because dune has no idea in which package any particular library must exist.
Ideally, we'd want 1. and 2. to run concurrently. One way to do that is to include additional metadata in opam files about the libraries that a package might provide. The following two constructs might be particularly useful:
x-findlib-library-mask: [
"bar" # the library will always be provided
"bar.*" # any number of libraries under the namespace bar. might be provided
"foo?" # the library foo might be optionally provided
]
This information will be sufficient for dune to only build the subset of packages that might contain a particular library.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how dune resolves libraries and sequences builds for packages in dune.lock and the local workspace. Review the proposed opam metadata, including library masks and optional libraries, and determine how it could safely support concurrent planning. Done means the two build phases can overlap without breaking library resolution, with coverage for the resulting build plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ocaml
- Domain
- build-system, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100