multiprocessing pool apply_async failure due to unable to pickle local dynamically created function

Bug report

This bug report is about multiprocessing module in standard library. When submit a local dynamically created function to pool executor in multiprocessing.Pool, it will failed. The details will be shown below.

Your environment

• CPython versions tested on: 3.10.6
• Operating system and architecture: Linux, x86_64

Details

Consider the following code snippet:

import multiprocessing
import pickle

pool = multiprocessing.Pool(4)


def error_callback(e):
    raise e


def go():
    for i in range(40):
        def hello(j):
            print(f"hello {i} {j}")

        pool.apply_async(hello, (i + 1,), error_callback=error_callback)

    pool.close()
    pool.join()


go()

The error is below:

Exception in thread Thread-2 (_handle_tasks):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 544, in _handle_tasks
    cache[job]._set(idx, (False, e))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 781, in _set
    self._error_callback(self._value)
  File "/.../xxx.py", line 8, in error_callback
    raise e
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 211, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'go.<locals>.hello'

analysis

Currently, the function in multiprocessing utilize pickle to transfer object between different process. When pickle.dumps() is applied to a function, only its reference information will be dumped. As a result, only global function which is defined in both sender and receiver end with same reference information will works.

The code object of function will not be dumped.

def hello():
    def hi(s):
        print(f"hi {s}")

    return pickle.dumps(hi)

hello()

Will cause same error:

Traceback (most recent call last):
  File "/.../.venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3398, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-8-a75d7781aaeb>", line 1, in <cell line: 1>
    hello()
  File "<ipython-input-7-3e50c99ad472>", line 4, in hello
    return pickle.dumps(hi)
AttributeError: Can't pickle local object 'hello.<locals>.hi'

potential solution

I am not meant to modify the behavior of pickle.dumps, but multiprocessing is supposed to utilize a enhanced version of pickle.

It is believed that the security issue is not significant in multiprocessing, because the serialized object which will be load at receiver end has already been executed in sender end. And the permissions of sender and receiver process is strictly the same.

So just for inspiration, marshal which can serialize code object can be mentioned here. Of course, a more complicated serializing function should be construct in multiprocessing which can rebuild a function at receiver end from scratch for local dynamically created function in sender end.

使用 multiprocessing.Pool 重现所提供的示例，并检查 multiprocessing/pool.py、connection.py 和 reduction.py 中的失败路径。将其与 pickle 对局部函数的处理进行比较；该 issue 提议对序列化进行广泛更改，但没有定义受支持的行为或具体的完成标准，因此请先与维护者确认范围。