multiprocessing pool apply_async failure due to unable to pickle local dynamically created function

Bug report

This bug report is about multiprocessing module in standard library. When submit a local dynamically created function to pool executor in multiprocessing.Pool, it will failed. The details will be shown below.

Your environment

• CPython versions tested on: 3.10.6
• Operating system and architecture: Linux, x86_64

Details

Consider the following code snippet:

import multiprocessing
import pickle

pool = multiprocessing.Pool(4)


def error_callback(e):
    raise e


def go():
    for i in range(40):
        def hello(j):
            print(f"hello {i} {j}")

        pool.apply_async(hello, (i + 1,), error_callback=error_callback)

    pool.close()
    pool.join()


go()

The error is below:

Exception in thread Thread-2 (_handle_tasks):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 544, in _handle_tasks
    cache[job]._set(idx, (False, e))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 781, in _set
    self._error_callback(self._value)
  File "/.../xxx.py", line 8, in error_callback
    raise e
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 211, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'go.<locals>.hello'

analysis

Currently, the function in multiprocessing utilize pickle to transfer object between different process. When pickle.dumps() is applied to a function, only its reference information will be dumped. As a result, only global function which is defined in both sender and receiver end with same reference information will works.

The code object of function will not be dumped.

def hello():
    def hi(s):
        print(f"hi {s}")

    return pickle.dumps(hi)

hello()

Will cause same error:

Traceback (most recent call last):
  File "/.../.venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3398, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-8-a75d7781aaeb>", line 1, in <cell line: 1>
    hello()
  File "<ipython-input-7-3e50c99ad472>", line 4, in hello
    return pickle.dumps(hi)
AttributeError: Can't pickle local object 'hello.<locals>.hi'

potential solution

I am not meant to modify the behavior of pickle.dumps, but multiprocessing is supposed to utilize a enhanced version of pickle.

It is believed that the security issue is not significant in multiprocessing, because the serialized object which will be load at receiver end has already been executed in sender end. And the permissions of sender and receiver process is strictly the same.

So just for inspiration, marshal which can serialize code object can be mentioned here. Of course, a more complicated serializing function should be construct in multiprocessing which can rebuild a function at receiver end from scratch for local dynamically created function in sender end.

提供された例を multiprocessing.Pool で再現し、multiprocessing/pool.py、connection.py、reduction.py の失敗経路を調査する。ローカル関数に対する pickle の処理と比較する。この issue はシリアライズの広範な変更を提案しているが、サポートする動作や具体的な完了基準を定義していないため、まず maintainer とスコープを確認する。