size limitation of UnbufferedCharStream and UnbufferedTokenStream in Java
- Dominant language
- Java
- Stars
- 19k
- Forks
- 3.5k
- PR merge metrics
- No merged PRs in 30d
Description
We use antlr4 UnbufferedCharStream and UnbufferedTokenStream (Java) to parse medium sized messages from (potentially) large files or streams with unknown size.
**antlr crashes after reading >2GB data (or >2*10e8 tokens).**
After digging into UnbufferedCharStream (and UnbufferedTokenStream) it became clear that this is because of the variable "currentCharIndex" holding the absolute character index as integer.
Unfortunately, changing this to long would affect the signature of "int index()" in "IntStream" which is used in a variety of files.
Do you plan to fix this in a future version?
As a quick-fix we removed the variable "currentCharIndex", return the variable "p" as index and return 0 in "getBufferStartIndex()" .
This "seems" to work in our case, but I don't oversee all consequences.
I see a potential danger in the functions "consume()" and "release()" which might reset "p" to 0 if no more marks are set.
Following scenario would probably cause a bug:
- fetch an index
- consume with 0 marks or release last mark
- do something with fetched index
I could not find such a scenario.
What do you think?
Contributor guide
Research direction
Start by reading UnbufferedCharStream and UnbufferedTokenStream, then inspect IntStream and the consume() and release() behavior described in the issue. Determine how indexes and marks must behave beyond the current integer limit, and define a compatible fix for streams exceeding 2GB without breaking existing callers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100