yapf errors with raw Python exception given null bytes in (invalid) source
- Dominant language
- Python
- Stars
- 14k
- Forks
- 904
- PR merge metrics
- No merged PRs in 30d
Description
This is in the long tail of bugs that there's really no reason you should care about, but given that I noticed it I figured I might as well mention it in case you cared.
Reproduction:
```
$ echo -e '\0' | yapf
Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/yapflib/pytree_utils.py", line 101, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/parse.py", line 116, in addtoken
ilabel = self.classify(type, value, context)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/parse.py", line 172, in classify
raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='\x00', context=('', (1, 0))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/yapflib/pytree_utils.py", line 107, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/parse.py", line 116, in addtoken
ilabel = self.classify(type, value, context)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/parse.py", line 172, in classify
raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='\x00', context=('', (1, 0))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.5.1/bin/yapf", line 11, in
sys.exit(run_main())
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/__init__.py", line 259, in run_main
sys.exit(main(sys.argv))
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/__init__.py", line 162, in main
verify=args.verify)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/yapflib/yapf_api.py", line 123, in FormatCode
tree = pytree_utils.ParseCodeToTree(unformatted_source)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/yapflib/pytree_utils.py", line 111, in ParseCodeToTree
ast.parse(code)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
ValueError: source code string cannot contain null bytes
```
I tested this running against a2f0342 under Python 3.5.1
Contributor guide
Assessment
This issue has not been assessed yet.