Instagram / Instagram/LibCST

Native parser crashes on pydoc_data.topics

Open
#980 0 comments 0 reactions 0 assignees View on GitHub
parsing
Dominant language
Python
Stars
1.9k
Forks
229
PR merge metrics
No merged PRs in 30d

Description

I am trying to parse in the source files of the standard library. When I get to Lib/pydoc_data/topics.py I encounter issues.
The native parser of LibCST causes stack overflow on this file. It can be parsed correctly with the pure parser.
I am not sure how to debug it, but this file contains thousands of lines of ConcatenatedString nodes, so I would guess that causes the problem. If I use the pure parser, the code will error anyway on the cst.code expression with a RecursionError.

Tested on windows 10 with LibCST 1.0.1 on Python 3.11.4.

Code to reproduce:
```python
import sys, os, faulthandler
from pathlib import Path
import libcst

# os.environ["LIBCST_PARSER_TYPE"] = "pure"
content = (Path(sys.base_prefix) / "Lib/pydoc_data/topics.py").read_text("utf-8")
faulthandler.enable()
cst: libcst.Module = libcst.parse_module(content)
print(cst.code)
```

Result of the run:
```
Windows fatal exception: stack overflow

Current thread 0x000020f8 (most recent call first):
File "D:\projects\testing\libcst_error\.venv\Lib\site-packages\libcst\_parser\entrypoints.py", line 55 in _parse
File "D:\projects\testing\libcst_error\.venv\Lib\site-packages\libcst\_parser\entrypoints.py", line 109 in parse_module
File "D:\projects\testing\libcst_error\main.py", line 8 in
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.