Performance issue in importModules{WithCache}
Nobody has claimed this yet.
- Dominant language
- Lean
- Stars
- 9.2k
- Forks
- 990
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 175
Description
This isn't a very common issue to run into, it only manifests when using importModules directly or in Lake's importModulesWithCache. In short, loading an olean file twice results in the second one missing the mmap fast path if the first is already in memory. This can easily happen when using importModulesWithCache because it only caches the exact same sequence of imports; multiple slightly different imports will very frequently collide on most of the loaded ModuleData objects, meaning that you lose the fast path on every olean file in Init and Lake when one lakefile uses import Lake and another one uses import Lean; import Lake. I hit this issue much more significantly when attempting something similar in mathport, because in this case I have to import Mathlib and startup time was in the 2 second range (per file), even when mathport itself has nothing to do.
My suggestion to fix this is to add a ModuleData cache so that loading the same olean twice doesn't needlessly map the file twice (since it's read-only anyway) and incur the cost of relocations on top of that.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read the importModules and importModulesWithCache entry points, then trace how ModuleData is loaded for repeated olean imports in Init and Lake. Use the described import sequences and mathport startup case to compare repeated loads. Done means repeated olean loads avoid losing the mmap fast path and do not incur duplicate mapping or relocation costs.
Written by the indexing model from the issue text.
Assessment
- Domain
- compilers, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100