ProcessPoolExecutor deadlocks against the import lock when the submitted callable is defined in the module being imported

Bug report

Bug description

ProcessPoolExecutor deadlocks silently — no output, no traceback, no exit — when all three of the following hold:

1. work is submitted from a module body, i.e. while that module is being imported;
2. the submitted callable is defined in that same module;
3. the module body blocks on the outcome, either on Future.result() or simply by leaving the with block, which calls shutdown(wait=True).

A timeout on Future.result() does not rescue it: when the timeout expires, __exit__ calls shutdown(wait=True), which joins the very thread that is stuck.

Minimal reproducer, mod_a.py:

from concurrent.futures import ProcessPoolExecutor


def square(x):
    return x * x


with ProcessPoolExecutor(max_workers=1) as ex:
    print(ex.submit(square, 3).result(timeout=30))

$ python -c "import mod_a"
   (hangs forever)

Control — byte-for-byte the same, except that the callable lives in another module.

worker.py:

def square(x):
    return x * x

mod_b.py:

from concurrent.futures import ProcessPoolExecutor

from worker import square

with ProcessPoolExecutor(max_workers=1) as ex:
    print(ex.submit(square, 3).result(timeout=30))

$ python -c "import mod_b"
9

The third ingredient is equally necessary: if the module body submits without waiting and the result is collected after the import has finished, there is no deadlock either.

Analysis

faulthandler.dump_traceback_later() on the hung process shows the cycle directly:

Thread 0x0001344c (most recent call first):
  File "<frozen importlib._bootstrap>", line 365 in acquire
  File "<frozen importlib._bootstrap>", line 471 in _lock_unlock_module
  File ".../Lib/multiprocessing/reduction.py", line 51 in dumps
  File ".../Lib/multiprocessing/queues.py", line 262 in _feed
  File ".../Lib/threading.py", line 995 in run
  File ".../Lib/threading.py", line 1044 in _bootstrap_inner
  File ".../Lib/threading.py", line 1015 in _bootstrap

Thread 0x00011d44 (most recent call first):
  File ".../Lib/threading.py", line 1095 in join
  File ".../Lib/concurrent/futures/process.py", line 851 in shutdown
  File ".../Lib/concurrent/futures/_base.py", line 647 in __exit__
  File ".../mod_a.py", line 9 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1023 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<string>", line 1 in <module>

• the main thread holds mod_a's per-module import lock for as long as the module body runs, and blocks inside that body waiting for the pool;
• the work item is pickled on the queue feeder thread (multiprocessing.queues.Queue._feed → multiprocessing.reduction.dumps);
• pickling square by reference calls __import__("mod_a"), which waits on that module's lock in importlib._bootstrap._lock_unlock_module.

Neither thread can move. Note that the cycle passes through a non-import resource — the queue and the future — so no amount of import-lock refinement can detect or break it.

What stands out is that the work item is serialized on the feeder thread rather than on the thread that called submit(). Had it been pickled in submit() — on the thread that already holds the module lock — this particular cycle could not form. There is precedent for moving work out of that thread for robustness reasons: bpo-31699 dealt with pickling errors in the same queue silently deadlocking the executor.

Prior art

• gh-51956 (bpo-7707), "multiprocess.Queue operations during import can lead to deadlocks", filed 2010, closed 2012 as fixed by a documentation note. I cannot find that note in the documentation today: Doc/library/multiprocessing.rst, Doc/library/threading.rst and Doc/library/concurrent.futures.rst on main say nothing about the import lock. It may well have been dropped when per-module import locks arrived in 3.3 (bpo-9260), which fixed most of this family — but not this case.
• gh-93580 is the mirror image, an import triggered while unpickling a result. It was closed as not planned, on the grounds that fork is unsafe in the presence of threads. That reasoning does not carry over here: this reproduces under spawn, and the whole deadlock lives inside the parent process.

Whatever the appetite for a code change, the silent hang seems worth at least a documented warning. Nothing surfaces — not a traceback, not a TimeoutError, not a broken-pool error — and the natural defensive measure, a timeout on result(), moves the hang from result() to shutdown() rather than avoiding it.

CPython versions tested on

3.13

Operating systems tested on

Windows

Nothing in the mechanism appears platform-specific — the feeder thread pickles under every start method — but I have only run it on Windows.

mod_a.py の例でハングを再現し、その後 Lib/concurrent/futures/process.py と、traceback で示されている multiprocessing の queue および reduction の経路を調査します。Doc/library/multiprocessing.rst、Doc/library/threading.rst、Doc/library/concurrent.futures.rst にある既存のガイダンスと比較します。この deadlock を防止するか、警告とその制限事項を明確に文書化できれば完了です。