python / python/cpython

multiprocessing.Process cannot be closed if the process is reaped elsewhere

Open
#94,661 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-multiprocessing type-bug
Dominant language
Python
Stars
77.2k
Forks
36k
PR merge metrics
PR metrics pending

Description

Here's a small reproducer where I start a multiprocessing.Process - then reap it with os.waitpid(

import os
import multiprocessing
import socket
import sys


ctx = multiprocessing.get_context("spawn")

def target(sock):
    with sock:
        sock.send(b"\x00")
        sock.recv(1)


def main():
    a, b = socket.socketpair()
    with a, b:
        with b:
            proc = ctx.Process(target=target, args=(b, ))
            proc.start()
        assert a.recv(1) == b"\x00"
        a.send(b"\x00")
        os.waitpid(proc.pid, 0)
        if sys.version_info >= (3, 7):
            proc.close()
        elif proc.is_alive():
            raise RuntimeError("proc={proc} is still running".format(proc=proc))


if __name__ == "__main__":
    sys.exit(main())

which results in:

 Traceback (most recent call last):
  File "/home/graingert/projects/distributed/demo.py", line 31, in <module>
    sys.exit(main())
  File "/home/graingert/projects/distributed/demo.py", line 25, in main
    proc.close()
  File "/home/graingert/miniconda3/lib/python3.9/multiprocessing/process.py", line 181, in close
    raise ValueError("Cannot close a process while it is still running. "
ValueError: Cannot close a process while it is still running. You should first call join() or terminate().

One issue is the call to self._popen.poll() suppresses the OSError of a reaped process:
https://github.com/python/cpython/blob/b6558d768f19584ad724be23030603280f9e6361/Lib/multiprocessing/popen_fork.py#L26-L31

I think this can be fixed by checking the state of the multiprocessing.Process.sentinel instead of using os.waitpid

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducer in the issue and inspect Lib/multiprocessing/popen_fork.py, especially the _popen.poll() handling, along with multiprocessing.Process.close(). Investigate how an externally reaped process is represented by its sentinel and compare that with waitpid-based state detection. Done means the reproducer can externally reap the child and then close the Process without the incorrect still-running ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.