antlr / antlr/antlr4

size limitation of UnbufferedCharStream and UnbufferedTokenStream in Java

Open
#2,897 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
19k
Forks
3.5k
PR merge metrics
No merged PRs in 30d

Description

We use antlr4 UnbufferedCharStream and UnbufferedTokenStream (Java) to parse medium sized messages from (potentially) large files or streams with unknown size.

**antlr crashes after reading >2GB data (or >2*10e8 tokens).**

After digging into UnbufferedCharStream (and UnbufferedTokenStream) it became clear that this is because of the variable "currentCharIndex" holding the absolute character index as integer.
Unfortunately, changing this to long would affect the signature of "int index()" in "IntStream" which is used in a variety of files.

Do you plan to fix this in a future version?

As a quick-fix we removed the variable "currentCharIndex", return the variable "p" as index and return 0 in "getBufferStartIndex()" .
This "seems" to work in our case, but I don't oversee all consequences.

I see a potential danger in the functions "consume()" and "release()" which might reset "p" to 0 if no more marks are set.
Following scenario would probably cause a bug:
- fetch an index
- consume with 0 marks or release last mark
- do something with fetched index

I could not find such a scenario.
What do you think?

Contributor guide

Open the contributing guide

Research direction

Start by reading UnbufferedCharStream and UnbufferedTokenStream, then inspect IntStream and the consume() and release() behavior described in the issue. Determine how indexes and marks must behave beyond the current integer limit, and define a compatible fix for streams exceeding 2GB without breaking existing callers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
compilers
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.