python / python/cpython

Unpickling Exceptions with keyword arguments in multiprocessing throws an error

未关闭
#120,810 14 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

@Eclips4 已经在做这个了。

开始于 2024年6月21日。

interpreter-core type-bug
主要语言
Python
星标
77.2k
派生
36k
PR 合并指标
PR 指标待抓取

描述

Bug report

Bug description:

I've tried using an Exception subclass with some fields and a simple constructor:

class Foo(Exception):
    foo: int
    bar: str

    def __init__(self, foo, bar):
        self.foo = foo
        self.bar = bar

When I construct and return this exception from a function invoked through ProcessPoolExecutor's map method, I get unpickling error:

(For simpler reproducer see https://github.com/python/cpython/issues/120810#issuecomment-2182228877)

import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor

class Foo(Exception):
    foo: int
    bar: str

    def __init__(self, foo, bar):
        self.foo = foo
        self.bar = bar

def work(x: str):
    print(mp.current_process().name)
    return Foo(foo=42, bar=x)

if __name__ == '__main__':
    with ProcessPoolExecutor() as pool:
        res = list(pool.map(work, ["a", "b", "c"]))

    print(res)

Traceback on 3.13.0b2 from official Docker image

concurrent.futures.process._RemoteTraceback: 
'''
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/concurrent/futures/process.py", line 424, in wait_result_broken_or_wakeup
    result_item = result_reader.recv()
  File "/usr/local/lib/python3.13/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
TypeError: Foo.__init__() missing 2 required positional arguments: 'foo' and 'bar'
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/check_mp.py", line 23, in <module>
    res = list(pool.map(work, ["a", "b", "c"]))
  File "/usr/local/lib/python3.13/concurrent/futures/process.py", line 623, in _chain_from_iterable_of_lists
    for element in iterable:
    ...<2 lines>...
            yield element.pop()
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Same error happens on 3.10, 3.12, and also on MacOS. Different fork method (fork, forkserver, spawn) all behave in the same way, both on Linux and MacOS.

However, if I construct Foo with positional arguments, there is no error:

import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor

class Foo(Exception):
    foo: int
    bar: str

    def __init__(self, foo, bar):
        self.foo = foo
        self.bar = bar

def work(x: str):
    print(mp.current_process().name)
    return Foo(42, x)

if __name__ == '__main__':
    with ProcessPoolExecutor() as pool:
        res = list(pool.map(work, ["a", "b", "c"]))

    print(res)

Output:

ForkProcess-1
ForkProcess-2
ForkProcess-3
[Foo(42, 'a'), Foo(42, 'b'), Foo(42, 'c')]

And, most strangely, if I remove Exception base class (leaving just class Foo():) there is no error with keyword arguments.

I've found only one similar issue: #103352, but that change does not fix my issue (tested on latest 3.13 beta).

Currently we workarounded this issue in our code by using positional arguments, but it was very annoying to debug this and getting to solution actually took us a couple of days.

CPython versions tested on:

3.10, 3.12, 3.13

Operating systems tested on:

Linux, macOS

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。