`preprocessFile` performance issues
- Dominant language
- Haskell
- Stars
- 1.7k
- Forks
- 750
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 28
Description
Hello! We've noticed that the `Preprocessing library for ...` part of setting up `cabal repl` in our very large package is taking a long time - somewhere between 10s and 60s. We have 10k+ modules and I don't think we have any files to preprocess by this mechanism (though we do have a few that use the `-pgmF` formatters - unsure if this is contributing).
I decided to dig into the source code and have identified some potential issues -
The code does [`for_ mods $ pre ...`](https://github.com/haskell/cabal/blob/63c486a1a65de599fa435c0cbf11ad85725f3c6c/Cabal/src/Distribution/Simple/PreProcess.hs?plain=1#L167-L168). This is probably the easiest place to get a win by introducing some concurrency.
The main part of the code in [`preprocessFile`](https://github.com/haskell/cabal/blob/63c486a1a65de599fa435c0cbf11ad85725f3c6c/Cabal/src/Distribution/Simple/PreProcess.hs?plain=1#L295-L310) does a file lookup for each of the suffixes in each of the directories in the search path. We have two search paths, and there are seven extensions: that's 14 calls to `doesFileExist` that will (in the common case) fail. I think this could be refactored to avoid the wasteful lookups in the more common case of "there's a `.hs` file".
Likewise, prepending `buildAsSrcLoc : searchLoc` is going to trigger an extra file lookup in the common case of "the module is in a `hs-source-dirs`." Doing `searchLoc ++ [buildAsSrcLoc]` should save a lookup.
In [`findFileCwdWithExtension' `](https://github.com/haskell/cabal/blob/63c486a1a65de599fa435c0cbf11ad85725f3c6c/Cabal/src/Distribution/Simple/Utils.hs?plain=1#L1321-L1326), we call `ordNub` on the search path and the extensions every time. While the two lists are very small and `ordNub` is efficient, we're hitting it ~20k times, meaning we're allocating 20k sets of 2 and 20k sets of 7. Using a `newtype Nubbed a = Nubbed [a]` with an `mkNubbed :: (Ord a) => [a] -> Nubbed a` would save this work from being repeated in a type-safe way, since the paths and extensions are pretty much shared in each invocation.
I'm happy to prepare a PR to do some of these performance improvements.
Contributor guide
Research direction
Start in Cabal/src/Distribution/Simple/PreProcess.hs, especially the linked module loop and preprocessFile, then inspect findFileCwdWithExtension' in Cabal/src/Distribution/Simple/Utils.hs. Measure the preprocessing portion of cabal repl on a large package before and after the changes. Done means the proposed lookups, repeated list processing, or module preprocessing no longer cause the reported slowdown without changing preprocessing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- build-system, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100