Instagram / Instagram/LibCST

Source code with CR and CRLF line endings causes PanicException

Open
#901 0 comments 0 reactions 0 assignees View on GitHub
bug parsing
Dominant language
Python
Stars
1.9k
Forks
229
PR merge metrics
No merged PRs in 30d

Description

Using LibCST 0.4.9

Reproduction code:
```py
import os
import libcst

os.environ["LIBCST_PARSER_TYPE"] = "native"
libcst.parse_module('x = 1\ry = 2\r\n')
```

Program output:
```
y = 2d '' panicked at 'byte index 14 is out of bounds of `x = 1
`', libcst/src/tokenizer/whitespace_parser.rs:370:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/tmp/libcst_repro.py", line 5, in
libcst.parse_module('x = 1\ry = 2\r\n')
File "/home/ubuntu/work/label-doconly-changes/.venvs/3.11/lib/python3.11/site-packages/libcst/_parser/entrypoints.py", line 109, in parse_module
result = _parse(
^^^^^^^
File "/home/ubuntu/work/label-doconly-changes/.venvs/3.11/lib/python3.11/site-packages/libcst/_parser/entrypoints.py", line 55, in _parse
return parse(source_str)
^^^^^^^^^^^^^^^^^
y = 2runtime.PanicException: byte index 14 is out of bounds of `x = 1
```

`RUST_BACKTRACE=full`:
```
y = 2d '' panicked at 'byte index 14 is out of bounds of `x = 1
`', libcst/src/tokenizer/whitespace_parser.rs:370:10
stack backtrace:
0: 0x7fe66d71bd50 - std::backtrace_rs::backtrace::libunwind::trace::h32eb3e08e874dd27
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x7fe66d71bd50 - std::backtrace_rs::backtrace::trace_unsynchronized::haa3f451d27bc11a5
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x7fe66d71bd50 - std::sys_common::backtrace::_print_fmt::h5b94a01bb4289bb5
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/backtrace.rs:66:5
3: 0x7fe66d71bd50 - ::fmt::hb070b7fa7e3175df
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/backtrace.rs:45:22
4: 0x7fe66d73638e - core::fmt::write::hd5207aebbb9a86e9
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/fmt/mod.rs:1202:17
5: 0x7fe66d719de5 - std::io::Write::write_fmt::h3bd699bbd129ab8a
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/io/mod.rs:1679:15
6: 0x7fe66d71d1b3 - std::sys_common::backtrace::_print::h7a21be552fdf58da
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/backtrace.rs:48:5
7: 0x7fe66d71d1b3 - std::sys_common::backtrace::print::ha85c41fe4dd80b13
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/backtrace.rs:35:9
8: 0x7fe66d71d1b3 - std::panicking::default_hook::{{closure}}::h04cca40023d0eeca
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:295:22
9: 0x7fe66d71ce9f - std::panicking::default_hook::haa3ca8c310ed5402
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:314:9
10: 0x7fe66d71d85a - std::panicking::rust_panic_with_hook::h7b190ce1a948faac
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:698:17
11: 0x7fe66d71d757 - std::panicking::begin_panic_handler::{{closure}}::hbafbfdc3e1b97f68
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:588:13
12: 0x7fe66d71c1fc - std::sys_common::backtrace::__rust_end_short_backtrace::hda93e5fef243b4c0
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/backtrace.rs:138:18
13: 0x7fe66d71d472 - rust_begin_unwind
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:584:5
14: 0x7fe66d52f743 - core::panicking::panic_fmt::h8d17ca1073d9a733
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:142:14
15: 0x7fe66d73967b - core::str::slice_error_fail_rt::h34f0d10f7cfd834f
16: 0x7fe66d52f947 - core::str::slice_error_fail::h84ffc2c9ae1a24ab
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/str/mod.rs:86:9
17: 0x7fe66d5ab98d - libcst_native::tokenizer::whitespace_parser::parse_simple_whitespace::hb4864ab9f19678c8
18: 0x7fe66d5aa752 - libcst_native::tokenizer::whitespace_parser::_parse_empty_lines::h8e0f556264181337
19: 0x7fe66d5aaaf6 - libcst_native::tokenizer::whitespace_parser::parse_empty_lines::h8c62c0b5d98da800
20: 0x7fe66d5aeb7c - ::inflate::hb49824cf292d8514
21: 0x7fe66d5ac328 - libcst_native::parse_module::h9164e22310aa42ea
22: 0x7fe66d67bd23 - std::panicking::try::hdca2d5fc2e011a6e
23: 0x7fe66d686362 - libcst_native::py::libcst_native::_::__pyfunction_parse_module::h6fd995afbdc03781
24: 0x54a7db -
25: 0x56d221 - PyObject_Call
26: 0x65686c -
27: 0x52ed4c - _PyObject_MakeTpCall
28: 0x54a6c5 - PyObject_Vectorcall
29: 0x53ca34 - _PyEval_EvalFrameDefault
30: 0x60fa74 -
31: 0x60eff7 - PyEval_EvalCode
32: 0x632a1b -
33: 0x62f014 -
34: 0x643605 -
35: 0x642cb4 - _PyRun_SimpleFileObject
36: 0x6429b7 - _PyRun_AnyFileObject
37: 0x63d449 - Py_RunMain
38: 0x5fef5d - Py_BytesMain
39: 0x7fe66ea83d90 - __libc_start_call_main
at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
40: 0x7fe66ea83e40 - __libc_start_main_impl
at ./csu/../csu/libc-start.c:392:3
41: 0x5fede5 - _start
42: 0x0 -
Traceback (most recent call last):
File "/tmp/libcst_repro.py", line 5, in
libcst.parse_module('x = 1\ry = 2\r\n')
File "/home/ubuntu/work/label-doconly-changes/.venvs/3.11/lib/python3.11/site-packages/libcst/_parser/entrypoints.py", line 109, in parse_module
result = _parse(
^^^^^^^
File "/home/ubuntu/work/label-doconly-changes/.venvs/3.11/lib/python3.11/site-packages/libcst/_parser/entrypoints.py", line 55, in _parse
return parse(source_str)
^^^^^^^^^^^^^^^^^
y = 2runtime.PanicException: byte index 14 is out of bounds of `x = 1
`
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.