ocaml / ocaml/dune

Improved Concurrency when building workspace and packages

Open
#8,650 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

package management
Dominant language
OCaml
Stars
1.9k
Forks
500
Avg merge
15h 21m
Merged PRs (30d)
277

Description

When using dune to build packages, the build plan must proceed as follows:

  1. Build all packages defined in dune.lock
  2. Build everything defined in the local workspace

These two steps must be done in sequence because to build something in the local workspace, dune must resolve the libraries being used. To resolve any particular library, dune needs to build all packages. This is because dune has no idea in which package any particular library must exist.

Ideally, we'd want 1. and 2. to run concurrently. One way to do that is to include additional metadata in opam files about the libraries that a package might provide. The following two constructs might be particularly useful:

x-findlib-library-mask: [
  "bar" # the library will always be provided
 "bar.*" # any number of libraries under the namespace bar. might be provided
 "foo?" # the library foo might be optionally provided
]

This information will be sufficient for dune to only build the subset of packages that might contain a particular library.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how dune resolves libraries and sequences builds for packages in dune.lock and the local workspace. Review the proposed opam metadata, including library masks and optional libraries, and determine how it could safely support concurrent planning. Done means the two build phases can overlap without breaking library resolution, with coverage for the resulting build plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
ocaml
Domain
build-system, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.