Minimal Package Dependencies

Background

When one writes a rule of the form:

(rule
 (deps (package foo))
 (action ..))

The rule will be executed once we build the (package foo) in _build/install. This is rather convenient feature, but it has its set of problems:

Generally, rules can observe stale artifacts in _build/install. So for example, a rule like:

(rule
 (action (run which foo)))

Will fail on succeed depending on whatever might be available in _build/install/$ctx/bin. Although this problem is completely mitigated by sandboxing, it comes with a performance cost.

Additionally, the way rules are implemented requires loading the artifacts for all packages before we can build anything in _build/install/$ctx. This is an annoying source of cycles as @ejgallego found out.

Proposal

My proposal is twofold:

First we improve the story with rule loading by setting up separate install rules for every package. With this change, it should be possible to build package x by using the META of package y if x depends on y. This is already need to make https://github.com/ocaml/dune/issues/8652 work.

Depending on multiple packages will now build the individual rules for every package and merge them all into a single directory tree like _build/install. This is quite similar to what's been proposed here for windows https://github.com/ocaml/dune/issues/8473

The second step is to remove all the paths _build/install that we include by default for all the rules. Such "install layout" paths will now only be visible if there are explicit (package ..) deps in the rule.

Start by reading the package dependency examples and the proposed behavior around _build/install, then review the related issues 8652 and 8473. The intended result is separate install rules for each package, with multiple package dependencies merged into one tree and install-layout paths visible only for explicit package dependencies.