Instagram / Instagram/LibCST

Recursion depth exceeded when applying MetadataWrapper on a very long concatenated string

Open
#1,178 2 comments 0 reactions 0 assignees View on GitHub
bug machinery
Dominant language
Python
Stars
1.9k
Forks
229
PR merge metrics
No merged PRs in 30d

Description

Running a simple roundtripper on every python file in the cpython repo's `Lib` dir found this interesting case.
_Aside: this is the only new case I found in my follow-up investigation from #1095._

[Here's a permalink to the current version of `pydoc_data/topics.py`](https://github.com/python/cpython/blob/04eb5c8db1e24cabd0cb81392bb2632c03be1550/Lib/pydoc_data/topics.py)

Attempting to roundtrip this file in my scratch env produces a huge trace of the form:

long trace (with sections snipped)

```
Traceback (most recent call last):
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 358, in deep_clone
cloned_fields[key] = tuple(_clone(v) for v in val)
^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'Dict' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 358, in deep_clone
cloned_fields[key] = tuple(_clone(v) for v in val)
^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'ConcatenatedString' object is not iterable

During handling of the above exception, another exception occurred:

... repeats many times ...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 358, in deep_clone
cloned_fields[key] = tuple(_clone(v) for v in val)
^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'ConcatenatedString' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 358, in deep_clone
cloned_fields[key] = tuple(_clone(v) for v in val)
^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'SimpleString' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/sirosen/_scratch/rt.py", line 21, in
_roundtrip_data(content)
File "/home/sirosen/_scratch/rt.py", line 12, in _roundtrip_data
wrapped_tree = libcst.MetadataWrapper(raw_tree)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/metadata/wrapper.py", line 146, in __init__
module = module.deep_clone()
^^^^^^^^^^^^^^^^^^^
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 358, in deep_clone
cloned_fields[key] = tuple(_clone(v) for v in val)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 358, in
cloned_fields[key] = tuple(_clone(v) for v in val)
^^^^^^^^^
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 105, in _clone
return val.deep_clone()
^^^^^^^^^^^^^^^^
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 358, in deep_clone
cloned_fields[key] = tuple(_clone(v) for v in val)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

... repeats many times ...

File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 362, in deep_clone
return type(self)(**cloned_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 6, in __init__
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/base.py", line 117, in __post_init__
self._validate()
File "/home/sirosen/_scratch/.venv/lib/python3.11/site-packages/libcst/_nodes/expression.py", line 592, in _validate
super(SimpleString, self)._validate()
RecursionError: maximum recursion depth exceeded
```

In case it's relevant, my no-op transformer is this body of code:

rt.py

```python
import sys

import libcst

class NoopTransformer(libcst.CSTTransformer):
pass

def _roundtrip_data(content: bytes) -> bytes:
raw_tree = libcst.parse_module(content)
wrapped_tree = libcst.MetadataWrapper(raw_tree)
tree = wrapped_tree.visit(NoopTransformer())
return tree.code.encode(tree.encoding)

fn = sys.argv[1]
with open(fn, "rb") as fp:
content = fp.read()

_roundtrip_data(content)
```

My guess would be that this is hard to solve because it probably requires unwinding a complex recursive construction into something non-recursive. Given that cpython can support this, I thought it was interesting enough to be worth reporting.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.