[RFC] Vendor in Vendor

Motivation

The main use case comes from Anil's experience with duniverse. Dune's composition model seems to break down for massive workspaces created by duniverse. Specifically, it does not work when vendoring dependencies with vendored dependencies. Let's consider using menhir as a dependency using duniverse. Here's a typical vendored directory:

vendor/
  menhir/
    vendor/
      fix.opam
  fix/
    fix.opam

dune does not like this because there are duplicate packages in the same workspace.

Problem

The problem of duplicate packages is the first one that pops up, but it's actually not the only one. Even if dune was to allow duplicate packages, it would be insufficient because both of these fix packages define the same fix public library. That's also problematic because dune expects all public libraries to live in the same namespace.

We could lift this restriction, but then how exactly should dune resolve the library fix?

Scopes in Dune

Dune has a good mechanism for resolving names and it's called a "Scope". Every library name in dune is resolved through a scope, and when a scope fails to resolve a name, the parent scope may resolve it. Currently, dune has three levels of scoping:

• Libraries defined in a single dune-project
• Public libraries in the entire workspace
• Installed libraries

Names are looked up in this order until one succeeds. Now it's apparent that the 2nd scope above is not fine grained enough to handle multiple public libraries.

Proposal

The simplest solution is then to introduce separate scopes for vendored directories. These two rules should be sufficient:

• A vendored directory should introduce its own scope
• Any public libraries defined in a vendored directory should not escape its parent scope.

In the first example, the fix library in ./vendor/menhir/vendor/fix lives in ./vendor/menhir/vendor and cannot escape past the scope defined in ./vendor.

So far, I've only discussed libraries. The same problems occur with executables and packages. Therefore, we'll need to generalize the concept of scopes to handle scoping for these entities as well. Currently, public binaries only live in a single scope. So it should be natural to generalize to a hierarchy like we do for libraries. There's a similar story for packages. Except that this will complicate (package foo) dependencies. A single _build/install/default environment will no longer be enough, and we'll need such an environment for every package scope.

Alternative Proposals

Earlier, my pet proposal to solve this issue was to introduce a first class for users to define such scopes and then possibly allow users to use fully qualified names. For example:

(scope
 (name vendored)
 (dirs *))

Then libraries in this scope can be referred to as lib@vendored for example. However, this was deemed to be as dumping complexity onto the users. It's not even clear there would be any other use cases for such a feature, so in the end we decided to go with fully inferred scopes.

Start by studying the scope-resolution model described in issue #3734 and how Dune currently handles vendored directories, public libraries, executables, and packages. Investigate the existing workspace and package-scope implementation before proposing a design. Done means nested vendored dependencies resolve without duplicate-package or public-name conflicts, including the stated package-environment concerns.