Creating `multiprocessing.Queues` operates strangely with `Process`'s created in classes
@Zheaoli y travaille déjà.
Depuis le 22/8/2024.
- Langage dominant
- Python
- Étoiles
- 77.2k
- Forks
- 35.9k
- Métriques de merge des PR
- Métriques de PR en attente
Description
Bug report
Bug description:
The following from the docs works:
from multiprocessing import Process, Queue
def f(q):
q.put([42, None, 'hello'])
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=(q,))
p.start()
print(q.get()) # prints "[42, None, 'hello']"
p.join()
The following also works:
from multiprocessing import Process, Queue
from time import sleep
def f(q):
pass
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=(q,))
p.start()
sleep(1)
This works just fine:
class Foo:
def __init__(self, q) -> None:
pass
if __name__ == '__main__':
q = Queue()
p = Process(target=Foo, args=(q,))
p.start()
sleep(1)
But the following does NOT work:
from multiprocessing import Process, Queue
from time import sleep
class Foo:
def __init__(self, q) -> None:
pass
class Bar:
def __init__(self) -> None:
q = Queue()
p = Process(target=Foo, args=(q,))
p.start()
if __name__ == '__main__':
Bar()
sleep(1)
Which now gives this traceback:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
self = reduction.pickle.load(from_parent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/synchronize.py", line 115, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
The following does work, though:
from multiprocessing import Process, Queue
from time import sleep
class Foo:
def __init__(self, q) -> None:
q.put("INIT COMPLETE")
class Bar:
def __init__(self) -> None:
q = Queue()
p = Process(target=Foo, args=(q,))
p.start()
q.get()
if __name__ == '__main__':
Bar()
sleep(1)
This should either be documented or fixed, preferably the latter. This behavior is strange and, quite frankly, unexpected. It took me quite a while to figure this out and fix my code but adding an init queue item is not ideal. Note that the following also breaks with the same error:
from multiprocessing import Process, Queue
from time import sleep
def foo(q) -> None:
pass
class Bar:
def __init__(self) -> None:
q = Queue()
p = Process(target=foo, args=(q,))
p.start()
if __name__ == '__main__':
Bar()
sleep(1)
Moving the Queue outside of the class (when calling it) also doesn't fix the issue:
from multiprocessing import Process, Queue
from time import sleep
def foo(q) -> None:
pass
class Bar:
def __init__(self, q) -> None:
p = Process(target=foo, args=(q,))
p.start()
if __name__ == '__main__':
Bar(Queue())
sleep(1)
But the following works just fine:
from multiprocessing import Process, Queue
from time import sleep
def foo(q) -> None:
pass
class Bar:
def __init__(self, q) -> None:
p = Process(target=foo, args=(q,))
p.start()
if __name__ == '__main__':
q = Queue()
Bar(q)
sleep(1)
This may be related to #116526 but I don't personally think this should be classified as the same issue. May be related to #23376 as well but this may still be a separate issue. Neither address this kind of behavior, however.
CPython versions tested on:
3.12
Operating systems tested on:
macOS
Guide de contribution
Ouvrir le guide de contribution
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Évaluation
Cette issue n'a pas encore été évaluée.