Instagram / Instagram/LibCST

"SyntaxError: invalid or missing encoding declaration" for (admittedly terrible) code that ast parses

Open
#1,078 0 comments 0 reactions 0 assignees View on GitHub
bug parsing
Dominant language
Python
Stars
1.9k
Forks
229
PR merge metrics
No merged PRs in 30d

Description

I can only apologise for this example, but the string `b"#\x80"` is accepted as valid Python by the ast module, `compile`, etc. but when passed to libcst gives the following error:

```
Traceback (most recent call last):
File "/Users/drmaciver/.pyenv/versions/3.12.0/lib/python3.12/tokenize.py", line 348, in find_cookie
line_string = line.decode('utf-8')
^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 1: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/Users/drmaciver/Projects/shrink-ray/.venv/lib/python3.12/site-packages/libcst/_parser/entrypoints.py", line 109, in parse_module
result = _parse(
^^^^^^^
File "/Users/drmaciver/Projects/shrink-ray/.venv/lib/python3.12/site-packages/libcst/_parser/entrypoints.py", line 44, in _parse
encoding, source_str = convert_to_utf8(source, partial=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/drmaciver/Projects/shrink-ray/.venv/lib/python3.12/site-packages/libcst/_parser/detect_config.py", line 125, in convert_to_utf8
_detect_encoding(source)
File "/Users/drmaciver/Projects/shrink-ray/.venv/lib/python3.12/site-packages/libcst/_parser/detect_config.py", line 49, in _detect_encoding
return py_tokenize_detect_encoding(BytesIO(source).readline)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/drmaciver/.pyenv/versions/3.12.0/lib/python3.12/tokenize.py", line 389, in detect_encoding
encoding = find_cookie(first)
^^^^^^^^^^^^^^^^^^
File "/Users/drmaciver/.pyenv/versions/3.12.0/lib/python3.12/tokenize.py", line 353, in find_cookie
raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.