`extra-source-files` glob behavior is very slow
- Dominant language
- Haskell
- Stars
- 1.7k
- Forks
- 750
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 28
Description
While diagnosing the very slow boot times in `cabal repl` for our codebase, I identified that a very large amount of time is being spent checking globs for `extra-source-files`. The log line that is emitted is [here](https://github.com/haskell/cabal/blob/63c486a1a65de599fa435c0cbf11ad85725f3c6c/Cabal/src/Distribution/Simple/Glob.hs?plain=1#L374).
I am actually having a *really* difficult time tracing the flow of this code, so I apologize if I've gone down some false paths or thrown out some red herrings.
When cabal [is checking the `extra-source-files`](https://github.com/haskell/cabal/blob/63c486a1a65de599fa435c0cbf11ad85725f3c6c/Cabal/src/Distribution/PackageDescription/Check.hs?plain=1#L457), it is doing so in a single threaded manner and is also interpreting each line as a glob.
Locally, I am observing about a 7-10ms time per entry - if I run `cabal repl --verbose="debug +timestamp"`, then I get this partial output:
```
1730308340.870 Expanding glob
'config/modelsFiles/abstract_category_mapping_rule.persistentmodels' in
directory '.'.
1730308340.875 Expanding glob
'config/modelsFiles/abstract_merchant_mapping_rule.persistentmodels' in
directory '.'.
...
1730308373.563 Expanding glob '.prettierrc' in directory './.'.
```
Per the timestamps, we're spending ~5-10ms in each glob, with a total elapsed time of 33 seconds (!!). There are ~1800 `extra-source-files` in our project, and ~10ms per would still only be 18s. Looking at the logs, they're each mentioned four times:
```
1730308346.904 Expanding glob 'config/modelsFiles/organization.persistentmodels' in directory
1730308354.357 Expanding glob 'config/modelsFiles/organization.persistentmodels' in directory
1730308363.024 Expanding glob 'config/modelsFiles/organization.persistentmodels' in directory
1730308370.972 Expanding glob 'config/modelsFiles/organization.persistentmodels' in directory
```
All `Expanding glob` are called in a block. The line immediately preceding the glob expansions is:
```
1730308340.639 Finalized package description:
```
And the message immediately after is:
```
1730308374.597 Dependency EventLog ==0.2.0.0: using EventLog-0.2.0.0
Dependency HUnit ==1.6.2.0: using HUnit-1.6.2.0
Dependency HaXml ==1.25.13: using HaXml-1.25.13
```
I need to put this down for right now, but I'll continue writing investigation notes here.
EDIT:
OK, I've identified that we're actually expanding each glob line in `extra-source-files` four times. Between only expanding the globs once and a fast-track for non-glob lines, I think this should be fixed up nicely.
Contributor guide
Research direction
Read Cabal/src/Distribution/Simple/Glob.hs at the logged expansion and Cabal/src/Distribution/PackageDescription/Check.hs at the extra-source-files handling. Trace why each entry is expanded four times; done means each line is expanded once, with non-glob entries avoiding unnecessary glob work, and cabal repl verbose timing confirms the improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- build-system, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100