Clarification on opam's `with-doc` and Dune's `(documentation …)` and `@doc` alias

Dear Dune/Opam maintainers,

Whilst hacking on ocaml-ci, I've discovered a shared misunderstanding about the with-doc opam dependency annotation, how dune build @doc works, and the Dune (documentation …) stanza.

We expected that we could annotate package dependency in the opam file that are only used for building the documentation with {with-doc}. For instance, md2mld or conf-graphviz to generate graphs to illustrate documentation pages. Then, we could use the (documentation …) stanza to attach the generated mld files, with rules to generate them and their dependencies. We expected these rules to be triggered only when dune build @doc was called.

ocaml-ci has a lint-doc steps that simply installs the package dependencies, odoc, and runs dune build @doc. As devs were adding with-doc annotations, I've added experimental support to install these packages too. The main ocaml-ci job however doesn't install these packages and run dune build @install @check @runtest. This main job started failing, because

All .mld files attached to a package will be included in the generated .install file for that package. They’ll be installed by opam.

so the mld files were dependencies of the install target, and couldn't be generated because the required packages hadn't been installed.

It seems a bit weird, but right now the with-doc annotation is only useful for the odoc package! because it's the only one Dune actually depends on for the @doc alias. It's not possible either to add (alias doc) to the rules because the mld files are also dependencies of @install.

If possible, I'd suggest either to stop installing mld files as part of @install or as dependencies of the install file, use with-doc
for packages used to generate documentation, and make @doc "understand" rules that generate documentation. If not, then we ought to be clearer and explain that with-doc and @doc are only used for the documentation processor, packages such as md2mld are regular (though it's conterintuitive) dependencies, and dune build @doc is solely a wrapper around odoc.

Start by comparing the issue's described behavior for dune build @doc, @install, (documentation …), and opam's with-doc annotation. Use the ocaml-ci lint-doc and main-job scenarios as the acceptance context; the result should either clarify the documented scope of these mechanisms or define the intended dependency behavior.