antlr / antlr/antlr4

Python2 runtime raises exception with unicode strings

Open
#999 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
19k
Forks
3.5k
PR merge metrics
No merged PRs in 30d

Description

Steps to reproduce:
- Create some grammar and run antlr4 -Dlanguage=Python2 Grammar.g4
- Now feed it a unicode string with high byte chars that doesn't match the grammar

The following exception is raised:

```
...
File ".../venv/local/lib/python2.7/site-packages/antlr4/Parser.py", line 347, in consume
self.getInputStream().consume()
File ".../venv/local/lib/python2.7/site-packages/antlr4/BufferedTokenStream.py", line 121, in consume
if self.sync(self.index + 1):
File ".../venv/local/lib/python2.7/site-packages/antlr4/BufferedTokenStream.py", line 134, in sync
fetched = self.fetch(n)
File ".../venv/local/lib/python2.7/site-packages/antlr4/BufferedTokenStream.py", line 146, in fetch
t = self.tokenSource.nextToken()
File ".../venv/local/lib/python2.7/site-packages/antlr4/Lexer.py", line 150, in nextToken
self.notifyListeners(e) # report error
File ".../uvea/venv/local/lib/python2.7/site-packages/antlr4/Lexer.py", line 303, in notifyListeners
msg = "token recognition error at: '" + self.getErrorDisplay(text) + "'"
File ".../venv/local/lib/python2.7/site-packages/antlr4/Lexer.py", line 310, in getErrorDisplay
buf.write(unicode(self.getErrorDisplayForChar(c)))
File ".../venv/local/lib/python2.7/site-packages/antlr4/Lexer.py", line 323, in getErrorDisplayForChar
return str(c)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0b95' in position 0: ordinal not in range(128)
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with a Python2-generated grammar and a high-byte unicode input, then inspect antlr4/Lexer.py around notifyListeners, getErrorDisplay, and getErrorDisplayForChar. Confirm that the same lexer error path reports the offending character without raising UnicodeEncodeError; the issue does not mention a test file to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.