python / python/cpython

concurrent.futures.ProcessPoolExecutor raises during shutdown

Open
#131,598 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stdlib topic-multiprocessing type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:
from concurrent.futures import ProcessPoolExecutor as Pool

def nope():
    pass

def mk_done(pool):
    def done(_fut):
        pool.shutdown(wait=True)
    return done

if __name__ == "__main__":
    pool = Pool()
    future = pool.submit(nope)
    future.add_done_callback(mk_done(pool))

ProcessPoolExecutor invokes future done callbacks from its manager thread. When one of those future callbacks invokes ProcessPoolExecutor.shutdown(), shut down fails with a RuntimeError: cannot join current thread because the implementation joins the manager thread without protecting against self-joins.

The relevant lines Lib/concurrent/futures/process.py:845-846 currently read:

if self._executor_manager_thread is not None and wait:
            self._executor_manager_thread.join()

but should probably be updated to:

t = self._executor_manager_thread
if t is not None and wait and threading.current_thread() != t:
            self._executor_manager_thread.join()

Invoking shutdown(False) avoids the problem but also makes it impossible to wait for pool completion. A more general solution would be to deprecate the wait parameter for shutdown, never waiting in that method, and add a separate method, say wait_for_shutdown, that uses a threading.Event to wake any waiting threads. Alas, correctly setting that Event is going to be a bit involved.

While I tested on 3.12, the bug is also present in 3.13 and current.

CPython versions tested on:

3.12

Operating systems tested on:

macOS

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided ProcessPoolExecutor reproduction, then inspect Lib/concurrent/futures/process.py around lines 845-846 and the shutdown entry point. Add coverage for shutdown(wait=True) from a future callback and verify that the callback completes without a self-join error while pool completion remains waitable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.