google / google/yapf

Unicode line separator und newline break parsing

Open
#949 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
14k
Forks
904
PR merge metrics
No merged PRs in 30d

Description

There are various line breaking characters, and these can be included in strings. Python can happily parse string literals that contain them. But YAPF fails to parse them, crashing with an EOS in a string literal.

Example tokens which crash YAPF are these:

- [U+0013](https://www.compart.com/en/unicode/U+0013): Device Control 3
- [U+001c](https://www.compart.com/en/unicode/U+001C): File Separator
- [U+0085](https://www.compart.com/de/unicode/U+0085): Next Line
- [U+2028](https://www.compart.com/en/unicode/U+2028): Line Separator

We might be able to escape these unicode literals with an escape code, but we'd prefer not to do it. Is there a way to make YAPF successfully parse the same files which the Python interpreter can also parse?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.