Unpickling Exceptions with keyword arguments in multiprocessing throws an error
@Eclips4 已經在處理了。
開始於 2024年6月21日。
- 主要語言
- Python
- 星號
- 77.2k
- 分支
- 36k
- PR 合併指標
- PR 指標待擷取
描述
Bug report
Bug description:
I've tried using an Exception subclass with some fields and a simple constructor:
class Foo(Exception):
foo: int
bar: str
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
When I construct and return this exception from a function invoked through ProcessPoolExecutor's map method, I get unpickling error:
(For simpler reproducer see https://github.com/python/cpython/issues/120810#issuecomment-2182228877)
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor
class Foo(Exception):
foo: int
bar: str
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def work(x: str):
print(mp.current_process().name)
return Foo(foo=42, bar=x)
if __name__ == '__main__':
with ProcessPoolExecutor() as pool:
res = list(pool.map(work, ["a", "b", "c"]))
print(res)
Traceback on 3.13.0b2 from official Docker image
concurrent.futures.process._RemoteTraceback:
'''
Traceback (most recent call last):
File "/usr/local/lib/python3.13/concurrent/futures/process.py", line 424, in wait_result_broken_or_wakeup
result_item = result_reader.recv()
File "/usr/local/lib/python3.13/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
TypeError: Foo.__init__() missing 2 required positional arguments: 'foo' and 'bar'
'''
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/data/check_mp.py", line 23, in <module>
res = list(pool.map(work, ["a", "b", "c"]))
File "/usr/local/lib/python3.13/concurrent/futures/process.py", line 623, in _chain_from_iterable_of_lists
for element in iterable:
...<2 lines>...
yield element.pop()
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
~~~~~~~~~~^^^^^^^^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Same error happens on 3.10, 3.12, and also on MacOS. Different fork method (fork, forkserver, spawn) all behave in the same way, both on Linux and MacOS.
However, if I construct Foo with positional arguments, there is no error:
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor
class Foo(Exception):
foo: int
bar: str
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def work(x: str):
print(mp.current_process().name)
return Foo(42, x)
if __name__ == '__main__':
with ProcessPoolExecutor() as pool:
res = list(pool.map(work, ["a", "b", "c"]))
print(res)
Output:
ForkProcess-1
ForkProcess-2
ForkProcess-3
[Foo(42, 'a'), Foo(42, 'b'), Foo(42, 'c')]
And, most strangely, if I remove Exception base class (leaving just class Foo():) there is no error with keyword arguments.
I've found only one similar issue: #103352, but that change does not fix my issue (tested on latest 3.13 beta).
Currently we workarounded this issue in our code by using positional arguments, but it was very annoying to debug this and getting to solution actually took us a couple of days.
CPython versions tested on:
3.10, 3.12, 3.13
Operating systems tested on:
Linux, macOS
貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
評估
這個 Issue 還沒有評估資料。