python / python/cpython

Unpickling Exceptions with keyword arguments in multiprocessing throws an error

Open
#120,810 14 comments 0 reactions 1 assignee View on GitHub

@Eclips4 is already working on this.

Since Jun 21, 2024.

interpreter-core type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

I've tried using an Exception subclass with some fields and a simple constructor:

class Foo(Exception):
    foo: int
    bar: str

    def __init__(self, foo, bar):
        self.foo = foo
        self.bar = bar

When I construct and return this exception from a function invoked through ProcessPoolExecutor's map method, I get unpickling error:

(For simpler reproducer see https://github.com/python/cpython/issues/120810#issuecomment-2182228877)

import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor

class Foo(Exception):
    foo: int
    bar: str

    def __init__(self, foo, bar):
        self.foo = foo
        self.bar = bar

def work(x: str):
    print(mp.current_process().name)
    return Foo(foo=42, bar=x)

if __name__ == '__main__':
    with ProcessPoolExecutor() as pool:
        res = list(pool.map(work, ["a", "b", "c"]))

    print(res)

Traceback on 3.13.0b2 from official Docker image

concurrent.futures.process._RemoteTraceback: 
'''
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/concurrent/futures/process.py", line 424, in wait_result_broken_or_wakeup
    result_item = result_reader.recv()
  File "/usr/local/lib/python3.13/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
TypeError: Foo.__init__() missing 2 required positional arguments: 'foo' and 'bar'
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/check_mp.py", line 23, in <module>
    res = list(pool.map(work, ["a", "b", "c"]))
  File "/usr/local/lib/python3.13/concurrent/futures/process.py", line 623, in _chain_from_iterable_of_lists
    for element in iterable:
    ...<2 lines>...
            yield element.pop()
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Same error happens on 3.10, 3.12, and also on MacOS. Different fork method (fork, forkserver, spawn) all behave in the same way, both on Linux and MacOS.

However, if I construct Foo with positional arguments, there is no error:

import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor

class Foo(Exception):
    foo: int
    bar: str

    def __init__(self, foo, bar):
        self.foo = foo
        self.bar = bar

def work(x: str):
    print(mp.current_process().name)
    return Foo(42, x)

if __name__ == '__main__':
    with ProcessPoolExecutor() as pool:
        res = list(pool.map(work, ["a", "b", "c"]))

    print(res)

Output:

ForkProcess-1
ForkProcess-2
ForkProcess-3
[Foo(42, 'a'), Foo(42, 'b'), Foo(42, 'c')]

And, most strangely, if I remove Exception base class (leaving just class Foo():) there is no error with keyword arguments.

I've found only one similar issue: #103352, but that change does not fix my issue (tested on latest 3.13 beta).

Currently we workarounded this issue in our code by using positional arguments, but it was very annoying to debug this and getting to solution actually took us a couple of days.

CPython versions tested on:

3.10, 3.12, 3.13

Operating systems tested on:

Linux, macOS

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.