Unpickling Exceptions with keyword arguments in multiprocessing throws an error
@Eclips4 ya está trabajando en esto.
Desde el 21/6/2024.
- Lenguaje dominante
- Python
- Estrellas
- 77.2k
- Forks
- 36k
- Métricas de merge de PR
- Métricas de PR pendientes
Descripción
Bug report
Bug description:
I've tried using an Exception subclass with some fields and a simple constructor:
class Foo(Exception):
foo: int
bar: str
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
When I construct and return this exception from a function invoked through ProcessPoolExecutor's map method, I get unpickling error:
(For simpler reproducer see https://github.com/python/cpython/issues/120810#issuecomment-2182228877)
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor
class Foo(Exception):
foo: int
bar: str
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def work(x: str):
print(mp.current_process().name)
return Foo(foo=42, bar=x)
if __name__ == '__main__':
with ProcessPoolExecutor() as pool:
res = list(pool.map(work, ["a", "b", "c"]))
print(res)
Traceback on 3.13.0b2 from official Docker image
concurrent.futures.process._RemoteTraceback:
'''
Traceback (most recent call last):
File "/usr/local/lib/python3.13/concurrent/futures/process.py", line 424, in wait_result_broken_or_wakeup
result_item = result_reader.recv()
File "/usr/local/lib/python3.13/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
TypeError: Foo.__init__() missing 2 required positional arguments: 'foo' and 'bar'
'''
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/data/check_mp.py", line 23, in <module>
res = list(pool.map(work, ["a", "b", "c"]))
File "/usr/local/lib/python3.13/concurrent/futures/process.py", line 623, in _chain_from_iterable_of_lists
for element in iterable:
...<2 lines>...
yield element.pop()
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
~~~~~~~~~~^^^^^^^^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Same error happens on 3.10, 3.12, and also on MacOS. Different fork method (fork, forkserver, spawn) all behave in the same way, both on Linux and MacOS.
However, if I construct Foo with positional arguments, there is no error:
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor
class Foo(Exception):
foo: int
bar: str
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def work(x: str):
print(mp.current_process().name)
return Foo(42, x)
if __name__ == '__main__':
with ProcessPoolExecutor() as pool:
res = list(pool.map(work, ["a", "b", "c"]))
print(res)
Output:
ForkProcess-1
ForkProcess-2
ForkProcess-3
[Foo(42, 'a'), Foo(42, 'b'), Foo(42, 'c')]
And, most strangely, if I remove Exception base class (leaving just class Foo():) there is no error with keyword arguments.
I've found only one similar issue: #103352, but that change does not fix my issue (tested on latest 3.13 beta).
Currently we workarounded this issue in our code by using positional arguments, but it was very annoying to debug this and getting to solution actually took us a couple of days.
CPython versions tested on:
3.10, 3.12, 3.13
Operating systems tested on:
Linux, macOS
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Evaluación
Este issue todavía no se ha evaluado.