python / python/cpython

Creating `multiprocessing.Queues` operates strangely with `Process`'s created in classes

オープン
#123,208 コメント 11 件 リアクション 0 件 担当者 1 名 GitHub で見る

@Zheaoli がすでに取り組んでいます。

2024年8月22日 から。

topic-multiprocessing type-bug
主要言語
Python
スター
77.2k
フォーク
35.9k
PR マージ指標
PR 指標を取得中

説明

Bug report

Bug description:

The following from the docs works:

from multiprocessing import Process, Queue

def f(q):
    q.put([42, None, 'hello'])

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    print(q.get())    # prints "[42, None, 'hello']"
    p.join()

The following also works:

from multiprocessing import Process, Queue
from time import sleep

def f(q):
    pass

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    sleep(1)

This works just fine:

class Foo:
    def __init__(self, q) -> None:
        pass

if __name__ == '__main__':
    q = Queue()
    p = Process(target=Foo, args=(q,))
    p.start()
    sleep(1)

But the following does NOT work:

from multiprocessing import Process, Queue
from time import sleep

class Foo:
    def __init__(self, q) -> None:
        pass

class Bar:
    def __init__(self) -> None:
        q = Queue()
        p = Process(target=Foo, args=(q,))
        p.start()

if __name__ == '__main__':
    Bar()
    sleep(1)

Which now gives this traceback:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/synchronize.py", line 115, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

The following does work, though:

from multiprocessing import Process, Queue
from time import sleep

class Foo:
    def __init__(self, q) -> None:
        q.put("INIT COMPLETE")

class Bar:
    def __init__(self) -> None:
        q = Queue()
        p = Process(target=Foo, args=(q,))
        p.start()
        q.get()

if __name__ == '__main__':
    Bar()
    sleep(1)

This should either be documented or fixed, preferably the latter. This behavior is strange and, quite frankly, unexpected. It took me quite a while to figure this out and fix my code but adding an init queue item is not ideal. Note that the following also breaks with the same error:

from multiprocessing import Process, Queue
from time import sleep

def foo(q) -> None:
    pass

class Bar:
    def __init__(self) -> None:
        q = Queue()
        p = Process(target=foo, args=(q,))
        p.start()

if __name__ == '__main__':
    Bar()
    sleep(1)

Moving the Queue outside of the class (when calling it) also doesn't fix the issue:

from multiprocessing import Process, Queue
from time import sleep

def foo(q) -> None:
    pass

class Bar:
    def __init__(self, q) -> None:
        p = Process(target=foo, args=(q,))
        p.start()

if __name__ == '__main__':
    Bar(Queue())
    sleep(1)

But the following works just fine:

from multiprocessing import Process, Queue
from time import sleep

def foo(q) -> None:
    pass

class Bar:
    def __init__(self, q) -> None:
        p = Process(target=foo, args=(q,))
        p.start()

if __name__ == '__main__':
    q = Queue()
    Bar(q)
    sleep(1)

This may be related to #116526 but I don't personally think this should be classified as the same issue. May be related to #23376 as well but this may still be a separate issue. Neither address this kind of behavior, however.

CPython versions tested on:

3.12

Operating systems tested on:

macOS

コントリビューションガイド

コントリビューションガイドを開く

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。