Crash when parsing function with unexpected await
- Dominant language
- Python
- Stars
- 14k
- Forks
- 904
- PR merge metrics
- No merged PRs in 30d
Description
## Current behaviour
Yapf crashes when attempting to format a function which contains an unexpected `await`.
```sh
cat > my_bad.py
```
```py
def run() -> None:
await do_thing()
^d
```
Install latest `yapf` from PyPI into a clean Python 3.12 venv and run it against `my_bad.py`:
```sh
pip install yapf==0.40.2
yapf my_bad.py
```
An `IndexError` is raised. Here's the full traceback:
```py
Traceback (most recent call last):
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/yapf_api.py", line 198, in FormatCode
tree = pytree_utils.ParseCodeToTree(unformatted_source)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/pytree/pytree_utils.py", line 113, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf_third_party/_ylib2to3/pgen2/driver.py", line 188, in parse_string
return self.parse_tokens(tokens, debug)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf_third_party/_ylib2to3/pgen2/driver.py", line 157, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf_third_party/_ylib2to3/pgen2/parse.py", line 230, in addtoken
return self._addtoken(ilabel, type, value, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf_third_party/_ylib2to3/pgen2/parse.py", line 313, in _addtoken
raise ParseError('bad input', type, value, context)
yapf_third_party._ylib2to3.pgen2.parse.ParseError: bad input: type=1, value='do_thing', context=(' ', (2, 10))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 241, in _FormatFile
reformatted_code, encoding, has_change = yapf_api.FormatFile(
^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/yapf_api.py", line 88, in FormatFile
reformatted_source, changed = FormatCode(
^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/yapf_api.py", line 201, in FormatCode
raise errors.YapfError(errors.FormatErrorMsg(e))
^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/errors.py", line 37, in FormatErrorMsg
return '{}:{}:{}: {}'.format(e.args[1][0], e.args[1][1], e.args[1][2], e.msg)
~~~~~~^^^
IndexError: tuple index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tmp.8dsxCLfnr9/venv312/bin/yapf", line 8, in
sys.exit(run_main())
^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 380, in run_main
sys.exit(main(sys.argv))
^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 139, in main
changed = FormatFiles(
^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 217, in FormatFiles
changed |= _FormatFile(filename, lines, style_config, no_local_style,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/__init__.py", line 251, in _FormatFile
raise errors.YapfError(errors.FormatErrorMsg(e))
^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmp.8dsxCLfnr9/venv312/lib/python3.12/site-packages/yapf/yapflib/errors.py", line 37, in FormatErrorMsg
return '{}:{}:{}: {}'.format(e.args[1][0], e.args[1][1], e.args[1][2], e.msg)
~~~~~~^^^
IndexError: tuple index out of range
```
Error is reproducible with Pythons 3.11 and 3.12. Also with latest version from git installed with `pip install -U git+https://github.com/google/yapf/` which gives the version:
```
yapf @ git+https://github.com/google/yapf/@6a92535967543aa979461fbe2da63a9b6f7dd595
```
## Expected behaviour
We would expect Yapf to do a nice job of warning of the bad syntax, rather than raising an exception.
For example, if we run with Python:
```sh
python my_bad.py
```
```
File "/tmp/tmp.8dsxCLfnr9/my_bad.py", line 2
await do_thing()
^^^^^^^^^^^^^^^^
SyntaxError: 'await' outside async function
```
And if we write "bad" Python, then Yapf usually warns us:
```sh
cat > other.py
```
```py
def thing()() -> None:
pass
^d
```
```sh
yapf other.py
```
```
yapf: other.py:1:12: expected ':'
```
Contributor guide
Assessment
This issue has not been assessed yet.