RecursionError in __getattr__ when running with multiprocessing
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 229
- PR merge metrics
- No merged PRs in 30d
Description
While trying to run a codemod against my codebase, I get lots of messages like this:
```
Traceback (most recent call last):
Traceback (most recent call last):
File ".../lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File ".../lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File ".../lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
File ".../lib/python3.8/multiprocessing/queues.py", line 358, in get
return _ForkingPickler.loads(res)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base File ".../lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File ".../lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File ".../lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
File ".../lib/python3.8/multiprocessing/queues.py", line 358, in get
return _ForkingPickler.loads(res)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
[Previous line repeated 989 more times]
RecursionError: maximum recursion depth exceeded
.py", line 386, in __getattr__
return getattr(self._matcher, key)
File ".../lib/python3.8/site-packages/libcst/matchers/_matcher_base.py", line 386, in __getattr__
return getattr(self._matcher, key)
[Previous line repeated 989 more times]
RecursionError: maximum recursion depth exceeded
```
Note that this is two interleaved tracebacks from parallel processing. Passing `-j1` to `codemod` prevents the error from occurring.
It looks like we've bumped into the [Surprising `__getattr__` recursion](https://nedbatchelder.com/blog/201010/surprising_getattr_recursion.html) caused by pickling and unpickling the arguments to get them into the multiprocessing workers. Code search [points](https://github.com/search?q=repo%3AInstagram%2FLibCST%20%22def%20__getattr__%22&type=code) to only two places where this needs to be fixed. It looks like adding
```python
if key == "_matcher":
raise AttributeError()
```
in both places should do the trick.
Contributor guide
Assessment
This issue has not been assessed yet.