Creating `multiprocessing.Queues` operates strangely with `Process`'s created in classes
@Zheaoli is already working on this.
Since Aug 22, 2024.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Bug report
Bug description:
The following from the docs works:
from multiprocessing import Process, Queue
def f(q):
q.put([42, None, 'hello'])
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=(q,))
p.start()
print(q.get()) # prints "[42, None, 'hello']"
p.join()
The following also works:
from multiprocessing import Process, Queue
from time import sleep
def f(q):
pass
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=(q,))
p.start()
sleep(1)
This works just fine:
class Foo:
def __init__(self, q) -> None:
pass
if __name__ == '__main__':
q = Queue()
p = Process(target=Foo, args=(q,))
p.start()
sleep(1)
But the following does NOT work:
from multiprocessing import Process, Queue
from time import sleep
class Foo:
def __init__(self, q) -> None:
pass
class Bar:
def __init__(self) -> None:
q = Queue()
p = Process(target=Foo, args=(q,))
p.start()
if __name__ == '__main__':
Bar()
sleep(1)
Which now gives this traceback:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
self = reduction.pickle.load(from_parent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/synchronize.py", line 115, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
The following does work, though:
from multiprocessing import Process, Queue
from time import sleep
class Foo:
def __init__(self, q) -> None:
q.put("INIT COMPLETE")
class Bar:
def __init__(self) -> None:
q = Queue()
p = Process(target=Foo, args=(q,))
p.start()
q.get()
if __name__ == '__main__':
Bar()
sleep(1)
This should either be documented or fixed, preferably the latter. This behavior is strange and, quite frankly, unexpected. It took me quite a while to figure this out and fix my code but adding an init queue item is not ideal. Note that the following also breaks with the same error:
from multiprocessing import Process, Queue
from time import sleep
def foo(q) -> None:
pass
class Bar:
def __init__(self) -> None:
q = Queue()
p = Process(target=foo, args=(q,))
p.start()
if __name__ == '__main__':
Bar()
sleep(1)
Moving the Queue outside of the class (when calling it) also doesn't fix the issue:
from multiprocessing import Process, Queue
from time import sleep
def foo(q) -> None:
pass
class Bar:
def __init__(self, q) -> None:
p = Process(target=foo, args=(q,))
p.start()
if __name__ == '__main__':
Bar(Queue())
sleep(1)
But the following works just fine:
from multiprocessing import Process, Queue
from time import sleep
def foo(q) -> None:
pass
class Bar:
def __init__(self, q) -> None:
p = Process(target=foo, args=(q,))
p.start()
if __name__ == '__main__':
q = Queue()
Bar(q)
sleep(1)
This may be related to #116526 but I don't personally think this should be classified as the same issue. May be related to #23376 as well but this may still be a separate issue. Neither address this kind of behavior, however.
CPython versions tested on:
3.12
Operating systems tested on:
macOS
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.