lark-parser / lark-parser/lark
grammar files get opened an unnecessary amount of times, causing an enormous loading time when creating a parser
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6k
- Forks
- 523
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 2
Description
it seems that, when using a FromPackageLoader object, a grammar file is opened and read from each time another grammar uses a rule that is imported from that former grammar. this means opening the same file over and over again, for each occurence of a rule contained in that file.
while this may not be noticeable for parsers that only use grammar files contained in the same directory (meaning no custom FromPackageLoader is necessary), it becomes highly problematic when using many FromPackageLoaders, as the time required to construct a parser goes up by an absurd amount.
by placing a print(resource_name) in the get_data() function of the python lib pkgutil.py, i was able to count how many times my grammar files were loaded each. for example, the common.lark grammar provided by lark gets opened 61 (!) times, one of my own grammars 25 times, another 16, etc.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing FromPackageLoader and its get_data() entry point, using the repeated loads of common.lark and the other grammar files as the reproduction signal. Confirm how often imported grammar resources are opened during parser construction, then verify that repeated requests no longer cause repeated file loads without changing grammar behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100