google / google/yapf

yapf errors with raw Python exception given null bytes in (invalid) source

Open
#330 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
14k
Forks
904
PR merge metrics
No merged PRs in 30d

Description

This is in the long tail of bugs that there's really no reason you should care about, but given that I noticed it I figured I might as well mention it in case you cared.

Reproduction:

```
$ echo -e '\0' | yapf
Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/yapflib/pytree_utils.py", line 101, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/parse.py", line 116, in addtoken
ilabel = self.classify(type, value, context)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/parse.py", line 172, in classify
raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='\x00', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/yapflib/pytree_utils.py", line 107, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/parse.py", line 116, in addtoken
ilabel = self.classify(type, value, context)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/lib2to3/pgen2/parse.py", line 172, in classify
raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='\x00', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.5.1/bin/yapf", line 11, in
sys.exit(run_main())
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/__init__.py", line 259, in run_main
sys.exit(main(sys.argv))
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/__init__.py", line 162, in main
verify=args.verify)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/yapflib/yapf_api.py", line 123, in FormatCode
tree = pytree_utils.ParseCodeToTree(unformatted_source)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/site-packages/yapf/yapflib/pytree_utils.py", line 111, in ParseCodeToTree
ast.parse(code)
File "/home/david/.pyenv/versions/3.5.1/lib/python3.5/ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
ValueError: source code string cannot contain null bytes
```

I tested this running against a2f0342 under Python 3.5.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.