"SyntaxError: invalid or missing encoding declaration" for (admittedly terrible) code that ast parses
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 229
- PR merge metrics
- No merged PRs in 30d
Description
I can only apologise for this example, but the string `b"#\x80"` is accepted as valid Python by the ast module, `compile`, etc. but when passed to libcst gives the following error:
```
Traceback (most recent call last):
File "/Users/drmaciver/.pyenv/versions/3.12.0/lib/python3.12/tokenize.py", line 348, in find_cookie
line_string = line.decode('utf-8')
^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 1: invalid start byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/Users/drmaciver/Projects/shrink-ray/.venv/lib/python3.12/site-packages/libcst/_parser/entrypoints.py", line 109, in parse_module
result = _parse(
^^^^^^^
File "/Users/drmaciver/Projects/shrink-ray/.venv/lib/python3.12/site-packages/libcst/_parser/entrypoints.py", line 44, in _parse
encoding, source_str = convert_to_utf8(source, partial=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/drmaciver/Projects/shrink-ray/.venv/lib/python3.12/site-packages/libcst/_parser/detect_config.py", line 125, in convert_to_utf8
_detect_encoding(source)
File "/Users/drmaciver/Projects/shrink-ray/.venv/lib/python3.12/site-packages/libcst/_parser/detect_config.py", line 49, in _detect_encoding
return py_tokenize_detect_encoding(BytesIO(source).readline)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/drmaciver/.pyenv/versions/3.12.0/lib/python3.12/tokenize.py", line 389, in detect_encoding
encoding = find_cookie(first)
^^^^^^^^^^^^^^^^^^
File "/Users/drmaciver/.pyenv/versions/3.12.0/lib/python3.12/tokenize.py", line 353, in find_cookie
raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration
```
Contributor guide
Assessment
This issue has not been assessed yet.