antlr / antlr/antlr4

An error occurred when antlr-parse is decoding the output.

Open
#4,282 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
19k
Forks
3.5k
PR merge metrics
No merged PRs in 30d

Description

When using antlr-parse to parse a file in utf-8 with encoding set to utf-8, an error occurred like below:

```text
Traceback (most recent call last): File "D:\scoop\apps\python39\current\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "D:\scoop\apps\python39\current\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "D:\scoop\apps\python39\current\Scripts\antlr4-parse.exe\__main__.py", line 7, in File "D:\scoop\apps\python39\current\lib\site-packages\antlr4_tool_runner.py", line 153, in interp err = err.decode("UTF-8")UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 198: invalid continuation byte
```

It seems the actual encoding of output is *gbk*, while it's decoded with *utf-8*. I've tried *chcp*(I'm using Windows), but the error remains.

An example grammar file is

```antlr4
grammar test;

// Comment: '//'~[\n\r]* -> skip;
EmptyLine: [\n\r]+ -> skip;
Space: [ \t] -> skip;
Token: ~[ \t\n\r]+;

program: Token*;
```

An example input file is

```text
任意的 Unicode 字符
```

Command is

```cmd
antlr4-parse test.g4 program -tree -encoding utf-8 test.txt
```

Contributor guide

Open the contributing guide

Research direction

Start with antlr4_tool_runner.py at the interp path reported at line 153, then reproduce the command `antlr4-parse test.g4 program -tree -encoding utf-8 test.txt` on Windows using the supplied grammar and Unicode input. Trace how the command output is decoded and verify that the error no longer raises UnicodeDecodeError for the example.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.