Extension to the `documentation` stanza to support new odoc 3 features

Hello dune people!

Since the release of odoc 3, the convention for building docs for opam-installed packages has been clarified and extended[^1]. Here are the bits to know:

[^1]: Let me stress that this has nothing to do with the rules for building documentation, it only has to do with installing the right files at the right places so that other drivers of odoc know how to build the documentation for the installed package!

1. Mld pages should still be installed in <switch-root>/doc/<pkgname>/odoc-pages/. However, the pages can now be nested inside subdirectories. The built doc will be faithful to this hierarchy (see this for details).
2. The hierarchy in <switch-root>/doc/<pkgname>/odoc-pages/ may now include non-mld files, which are then considered assets. See for instance this line to install an image asset in odoc's doc. (See this)
3. The docs may refer to additional packages and libraries than the actual package's dependencies, for instance to refer to the docs of a package using it (eg, odoc referencing odig's docs). This is specified in a file (in sexp syntax) that has to be installed as <switch-root>/doc/<pkgname>/odoc-config.sexp. See this and this and this for more detail, and the file for odoc and the rule to install it for an example.
4. In order for ocaml.org's doc CI to be able to install all needed packages, the opam file of a package should also include all packages it depends upon, under the x-extra-doc-deps field name. See this for details. For instance, here is odoc specifying its doc dependencies.

Although it's possible to install assets, config file and subdirectories using the install stanza, I believe the (documentation ...) stanza should be extended to properly support those new features. Here is a proposition:

Documentation dependencies

Point 3. and 4. above: Adding a (doc-depends (libraries ...) (packages ...)) stanza in the package stanza of dune-project file would allow to generate both the x-extra-doc-deps opam field, and the odoc-config.sexp file. For instance, both:

• this part of odoc's opam file would be generated with:
• the odoc-config.sexp file and the rule to install it

would be replaced by the following doc-depends stanza in `dune-project:

(package
 (name odoc)
 ...
 (doc-depends
  (libraries
   fmt
   ...)
  (packages
   odig
   cmdliner
   (odoc-driver (= version))
   (sherlodoc (= version)))
  ...))

Documentation hierarchy

(Point 1. and 2. above) I think the most natural way to define your installed doc's hierarchy is to define it as being the same as the one in your source: The hierarchy:

doc/dune           # contains a documentation stanza
doc/index.mld
doc/asset.jpg
doc/examples.mld
doc/tutorial/index.mld
doc/tutorial/tuto1.mld
doc/tutorial/tuto2.mld

would generate the installation of:

<root>/doc/<pkgname>/odoc-pages/index.mld
<root>/doc/<pkgname>/odoc-pages/asset.jpg
<root>/doc/<pkgname>/odoc-pages/examples.mld
<root>/doc/<pkgname>/odoc-pages/tutorial/index.mld
<root>/doc/<pkgname>/odoc-pages/tutorial/tuto1.mld
<root>/doc/<pkgname>/odoc-pages/tutorial/tuto2.mld

However:

• Dune seems to mostly work in a directory (apart from include_subdirs)
• We need to be able to specify which files and directories we want

I propose to add several stanza to (documentation ...):

• (files ...) which can contain a list of (file <filename>) (or simply <filename>), (glob_files <glob>), (glob_files_rec <glob>), (source_tree <dir>) (that is, the relevant subset of dependency specification for rules). I guess the dune file in which the documentation stanza is defined is not included, unless explicitly specified.
• (path <path>) allows to install all mld files under <root>/doc/<pkgname>/odoc-pages/<path>/.

Omitting (path ...) installs files directly in odoc-pages/.
Omitting (files ...) and (mld_files ...) results in including everything, including subdirs. I believe in earlier dune-lang versions, the previous behaviour of including only mld files from the current directory is restored!

Conclusion

I'm happy to join the next dune dev meeting to discuss that.
And once the design has been agreed on, I'll be happy to start implementing it!

Start by reading the documentation stanza in doc/dune and the related package metadata in dune-project and odoc.opam. Compare these with doc/odoc-config.sexp and the linked odoc examples. Done means the agreed design supports documentation dependencies, nested pages and assets, and the required installed metadata without relying on ad hoc install rules.