Instagram / Instagram/LibCST

RecursionError in __getattr__ when running with multiprocessing

Open
#907 0 comments 2 reactions 0 assignees View on GitHub
bug machinery
Dominant language
Python
Stars
1.9k
Forks
229
PR merge metrics
No merged PRs in 30d

Description

While trying to run a codemod against my codebase, I get lots of messages like this:

```
Traceback (most recent call last):
Traceback (most recent call last):
File ".../lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File ".../lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File ".../lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
File ".../lib/python3.8/multiprocessing/queues.py", line 358, in get
return _ForkingPickler.loads(res)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base File ".../lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File ".../lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File ".../lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
File ".../lib/python3.8/multiprocessing/queues.py", line 358, in get
return _ForkingPickler.loads(res)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
[Previous line repeated 989 more times]
RecursionError: maximum recursion depth exceeded
.py", line 386, in __getattr__
return getattr(self._matcher, key)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
[Previous line repeated 989 more times]
RecursionError: maximum recursion depth exceeded
```

Note that this is two interleaved tracebacks from parallel processing. Passing `-j1` to `codemod` prevents the error from occurring.

It looks like we've bumped into the [Surprising `__getattr__` recursion](https://nedbatchelder.com/blog/201010/surprising_getattr_recursion.html) caused by pickling and unpickling the arguments to get them into the multiprocessing workers. Code search [points](https://github.com/search?q=repo%3AInstagram%2FLibCST%20%22def%20__getattr__%22&type=code) to only two places where this needs to be fixed. It looks like adding

```python
if key == "_matcher":
raise AttributeError()
```

in both places should do the trick.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.