Module listings in the presence of Backpack

Let's talk about package index pages; you know, the thing you see when you navigate to a package page like https://hackage.haskell.org/package/time

Currently, Hackage duplicates the algorithm for rendering a module tree with Haddock. Here is Hackage's copy of ModuleForest: https://github.com/haskell/hackage-server/blob/master/Distribution/Server/Packages/ModuleForest.hs and here is Haddock's copy https://github.com/haskell/haddock/blob/master/haddock-api/src/Haddock/ModuleTree.hs Haddock generated its index from the list of modules that would have been passed to GHC to compile; Hackage generates it from a flattened package description. For the longest time, this didn't really cause any problems because the module tree format didn't ever really change (just read it off of `exposed-modules`).

Unfortunately, I recently added two new features to Cabal which have broken this assumption:

* Reexported modules means that exposed modules are not limited to `exposed-modules`; they may also be defined in `reexported-modules`. This has meant that Hackage does not actually display reexported modules. To make matters worse, if reexported modules are implemented as links to the reexported module in another package (a reasonable strategy) this logic must be duplicated in both Haddock and Hackage. And the cherry on top: Hackage doesn't even have enough information to actually compute this correctly. More context at https://github.com/haskell/haddock/issues/563

* Signatures can be inherited from packages, and we don't make it mandatory for a user to specify all signatures in their Cabal file (we can infer it when we are configuring the package.) But there will be Haddock documentation for every real signature in the package. Once again, Hackage has no way of figuring this out when it is generating its index.

I want to argue that it is *morally wrong* for Hackage to be in the business of generating the module tree that connects to the documentation in question. Not only does Hackage not have enough information to do it properly, Haddock is already in the business of computing it.

So, I think the correct model for Haddock to generate the HTML for the module tree, and then Hackage to transclude it into the index page, in much the same way Hackage transcludes the README into the index page.

Here's the annoying thing though: current Haddock, when invoked by Cabal to build documentation for Hackage, doesn't actually generate the index, because it is passed the `--use-contents=` flag which stops Haddock from generating the index. And even if it did generate the index, the HTML we are interested in is embedded inside a div inside the index HTML proper.

So, it would seem, to properly fix this, we would have to modify Haddock to generate a module tree which Hackage can transclude in. This is not great, since Haddock is on the same release cycle as GHC. We can always fall back on the old code if Haddock doesn't support this, but this means we can't really delete anything.

An alternative, pragmatic solution is to continue limping along with some less-than-perfect-but-good-enough module tree generation code in Hackage. For example, to get accurate signature listing, you'd simply be forced to list every signature explicitly in your Cabal file to help Cabal out. reexported-modules would remain a disaster for the forseeable future, or perhaps we can levy a Haddock level fix to copy in the documentation.

What do people think?

Start by reading Distribution/Server/Packages/ModuleForest.hs in Hackage and haddock-api/src/Haddock/ModuleTree.hs in Haddock, then examine how --use-contents= affects generated indexes. Decide and implement a model for an accurate, transcludable module tree that handles reexported modules and inherited signatures, including the fallback behavior described in the issue.

Written by the indexing model from the issue text.