Reduce memory usage when parsing lots of different inputs.
- Dominant language
- Java
- Stars
- 19k
- Forks
- 3.5k
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I am using antlr cppruntime to build a fuzzer, which needs to parse lots of different inputs of a grammar. However, the memory usage is too high (over 50G) after parsing millions of inputs. I can confirm that there are no memory leakage on my parts. I profile the memory usage and they are mostly in `ParserATNSimulator`.
I checked the issues (https://github.com/antlr/antlr4/issues/2182, https://github.com/antlr/antlr4/issues/499), which don't solve my problems: The memory usage is still high and clearing the cache makes the parser really slow sometime (completely stuck for a few minutes).
Every time I parse a input, I create a new parser as below:
```cpp
ANTLRInputStream input(input_program);
XXXLexer lexer(&input);
CommonTokenStream tokens(&lexer);
tokens.fill();
XXXParser parser(&tokens);
```
Are there any ways to limit the memory usage? Any suggestions are appreciated!
Contributor guide
Research direction
Start by profiling ParserATNSimulator while reproducing the C++ parsing loop shown in the issue across many inputs. Review issues 2182 and 499 to understand existing cache-clearing approaches. Done means identifying a concrete way to bound memory growth while preserving acceptable parsing performance, then validating it on the reported workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100