antlr / antlr/antlr4

Reduce memory usage when parsing lots of different inputs.

Open
#4,232 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
19k
Forks
3.5k
PR merge metrics
No merged PRs in 30d

Description

Hi,

I am using antlr cppruntime to build a fuzzer, which needs to parse lots of different inputs of a grammar. However, the memory usage is too high (over 50G) after parsing millions of inputs. I can confirm that there are no memory leakage on my parts. I profile the memory usage and they are mostly in `ParserATNSimulator`.

I checked the issues (https://github.com/antlr/antlr4/issues/2182, https://github.com/antlr/antlr4/issues/499), which don't solve my problems: The memory usage is still high and clearing the cache makes the parser really slow sometime (completely stuck for a few minutes).

Every time I parse a input, I create a new parser as below:
```cpp
ANTLRInputStream input(input_program);
XXXLexer lexer(&input);
CommonTokenStream tokens(&lexer);
tokens.fill();
XXXParser parser(&tokens);
```

Are there any ways to limit the memory usage? Any suggestions are appreciated!

Contributor guide

Open the contributing guide

Research direction

Start by profiling ParserATNSimulator while reproducing the C++ parsing loop shown in the issue across many inputs. Review issues 2182 and 499 to understand existing cache-clearing approaches. Done means identifying a concrete way to bound memory growth while preserving acceptable parsing performance, then validating it on the reported workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.