Instagram / Instagram/LibCST

MatchIfTrue lambdas used in decorator matchers in VisitorBasedCodemodCommand hit pickle errors on large codebase

Open
#798 4 comments 1 reaction 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
1.9k
Forks
229
PR merge metrics
No merged PRs in 30d

Description

In a subclass of `VisitorBasedCodemodCommand`, I have code that creates a `@matchers.visit` with a MatchIfTrue lambda that throws a lambda pickling error when the codemod is run on a sufficiently large codebase. Running it on a file at a time is fine, because the `multiprocessing` work doesn't kick in.

Probably it deserves a call out in the docs of `MatchIfTrue` to use named functions instead of lambdas. I'm not sure what other solutions are available.

The command in question:

```python
def func_def_has_request_param_matcher() -> m.FunctionDef:
return m.FunctionDef(
params=m.Parameters(
params=m.MatchIfTrue(
lambda params: any(p.name.value == "request" for p in params)
),
)
)

class ReproCommand(VisitorBasedCodemodCommand):
DESCRIPTION = (
"Do cool thing."
)

@staticmethod
def add_args(_arg_parser: argparse.ArgumentParser) -> None:
return

@m.visit(func_def_has_request_param_matcher())
def _mark_func_def_includes_request_param(self, node: cst.FunctionDef) -> None:
self.context.scratch["cool-scratch"] = node
```

The error in question

$ python -m libcst.tool codemod commands.repro.ReproCommand src
Calculating full-repo metadata...
Executing codemod...
Traceback (most recent call last):
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 839, in
main(os.environ.get("LIBCST_TOOL_COMMAND_NAME", "libcst.tool"), sys.argv[1:])
File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 834, in main
return lookup.get(args.action or None, _invalid_command)(proc_name, command_args)
File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 572, in _codemod_impl
result = parallel_exec_transform_with_prettyprint(
File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/codemod/_cli.py", line 638, in parallel_exec_transform_with_prettyprint
for result in p.imap_unordered(
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 448, in
return (item for chunk in result for item in chunk)
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks
put(task)
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'func_def_has_request_param_matcher..'

Inlining the matcher creation gets a slightly different error

$ python -m libcst.tool codemod commands.repro.ReproCommand src
Calculating full-repo metadata...
Executing codemod...
Traceback (most recent call last):
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 839, in
main(os.environ.get("LIBCST_TOOL_COMMAND_NAME", "libcst.tool"), sys.argv[1:])
File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 834, in main
return lookup.get(args.action or None, _invalid_command)(proc_name, command_args)
File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/tool.py", line 572, in _codemod_impl
result = parallel_exec_transform_with_prettyprint(
File "/Users/jeff.hodges/src/github.com/color/color/local/virtualenv3/lib/python3.8/site-packages/libcst/codemod/_cli.py", line 638, in parallel_exec_transform_with_prettyprint
for result in p.imap_unordered(
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 448, in
return (item for chunk in result for item in chunk)
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks
put(task)
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/Users/jeff.hodges/.asdf/installs/python/3.8.13/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle at 0x10205b1f0>: attribute lookup ReproCommand. on projects.codemods.pycodemods.commands.repro failed

(Tangentially, if there's a better way to do this "match if any Node in this Sequence matches this pattern", I'm down to be told what that is)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.