Python2 runtime raises exception with unicode strings
- Dominant language
- Java
- Stars
- 19k
- Forks
- 3.5k
- PR merge metrics
- No merged PRs in 30d
Description
Steps to reproduce:
- Create some grammar and run antlr4 -Dlanguage=Python2 Grammar.g4
- Now feed it a unicode string with high byte chars that doesn't match the grammar
The following exception is raised:
```
...
File ".../venv/local/lib/python2.7/site-packages/antlr4/Parser.py", line 347, in consume
self.getInputStream().consume()
File ".../venv/local/lib/python2.7/site-packages/antlr4/BufferedTokenStream.py", line 121, in consume
if self.sync(self.index + 1):
File ".../venv/local/lib/python2.7/site-packages/antlr4/BufferedTokenStream.py", line 134, in sync
fetched = self.fetch(n)
File ".../venv/local/lib/python2.7/site-packages/antlr4/BufferedTokenStream.py", line 146, in fetch
t = self.tokenSource.nextToken()
File ".../venv/local/lib/python2.7/site-packages/antlr4/Lexer.py", line 150, in nextToken
self.notifyListeners(e) # report error
File ".../uvea/venv/local/lib/python2.7/site-packages/antlr4/Lexer.py", line 303, in notifyListeners
msg = "token recognition error at: '" + self.getErrorDisplay(text) + "'"
File ".../venv/local/lib/python2.7/site-packages/antlr4/Lexer.py", line 310, in getErrorDisplay
buf.write(unicode(self.getErrorDisplayForChar(c)))
File ".../venv/local/lib/python2.7/site-packages/antlr4/Lexer.py", line 323, in getErrorDisplayForChar
return str(c)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0b95' in position 0: ordinal not in range(128)
```
Contributor guide
Research direction
Reproduce the failure with a Python2-generated grammar and a high-byte unicode input, then inspect antlr4/Lexer.py around notifyListeners, getErrorDisplay, and getErrorDisplayForChar. Confirm that the same lexer error path reports the offending character without raising UnicodeEncodeError; the issue does not mention a test file to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100