haskell / haskell/cabal

`extra-source-files` glob behavior is very slow

Open
#10,495 1 comment 0 reactions 0 assignees View on GitHub
re: extra-source-files type: user-question
Dominant language
Haskell
Stars
1.7k
Forks
750
Avg merge
4d 3h
Merged PRs (30d)
28

Description

While diagnosing the very slow boot times in `cabal repl` for our codebase, I identified that a very large amount of time is being spent checking globs for `extra-source-files`. The log line that is emitted is [here](https://github.com/haskell/cabal/blob/63c486a1a65de599fa435c0cbf11ad85725f3c6c/Cabal/src/Distribution/Simple/Glob.hs?plain=1#L374).

I am actually having a *really* difficult time tracing the flow of this code, so I apologize if I've gone down some false paths or thrown out some red herrings.

When cabal [is checking the `extra-source-files`](https://github.com/haskell/cabal/blob/63c486a1a65de599fa435c0cbf11ad85725f3c6c/Cabal/src/Distribution/PackageDescription/Check.hs?plain=1#L457), it is doing so in a single threaded manner and is also interpreting each line as a glob.

Locally, I am observing about a 7-10ms time per entry - if I run `cabal repl --verbose="debug +timestamp"`, then I get this partial output:

```
1730308340.870 Expanding glob
'config/modelsFiles/abstract_category_mapping_rule.persistentmodels' in
directory '.'.
1730308340.875 Expanding glob
'config/modelsFiles/abstract_merchant_mapping_rule.persistentmodels' in
directory '.'.

...

1730308373.563 Expanding glob '.prettierrc' in directory './.'.
```

Per the timestamps, we're spending ~5-10ms in each glob, with a total elapsed time of 33 seconds (!!). There are ~1800 `extra-source-files` in our project, and ~10ms per would still only be 18s. Looking at the logs, they're each mentioned four times:

```
1730308346.904 Expanding glob 'config/modelsFiles/organization.persistentmodels' in directory
1730308354.357 Expanding glob 'config/modelsFiles/organization.persistentmodels' in directory
1730308363.024 Expanding glob 'config/modelsFiles/organization.persistentmodels' in directory
1730308370.972 Expanding glob 'config/modelsFiles/organization.persistentmodels' in directory
```

All `Expanding glob` are called in a block. The line immediately preceding the glob expansions is:

```
1730308340.639 Finalized package description:
```

And the message immediately after is:

```
1730308374.597 Dependency EventLog ==0.2.0.0: using EventLog-0.2.0.0
Dependency HUnit ==1.6.2.0: using HUnit-1.6.2.0
Dependency HaXml ==1.25.13: using HaXml-1.25.13
```

I need to put this down for right now, but I'll continue writing investigation notes here.

EDIT:

OK, I've identified that we're actually expanding each glob line in `extra-source-files` four times. Between only expanding the globs once and a fast-track for non-glob lines, I think this should be fixed up nicely.

Contributor guide

Open the contributing guide

Research direction

Read Cabal/src/Distribution/Simple/Glob.hs at the logged expansion and Cabal/src/Distribution/PackageDescription/Check.hs at the extra-source-files handling. Trace why each entry is expanded four times; done means each line is expanded once, with non-glob entries avoiding unnecessary glob work, and cabal repl verbose timing confirms the improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
build-system, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.