python / python/cpython

Inconsistent KeyboardInterrupt Handling in multiprocessing due to Context Manager Order

Open
#119,104 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-multiprocessing type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

When using multiprocessing.Manager with concurrent.futures.ProcessPoolExecutor, there is a particular ordering of the context managers that results in multiprocessing.managers.py hanging in the method serve_forever, forever. This can be triggered by raising a KeyboardInterrupt while the child process(es), here something, are busy. The ordering that leads to the bug is: Manager first, then ProcessPoolExecutor inside.

import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from time import sleep


def something():
    sleep(10)

# Uncomment one of the following blocks

### Works correctly:

# # Interrupt with CTRL+C while `something` is busy
# # Takes just one KeyboardInterrupts to terminate fully
# with ProcessPoolExecutor() as executor:
#     futures = []
#     with multiprocessing.Manager() as manager:
#         futures.append(executor.submit(something))
#
#         for f in futures:
#             f.result()

### Doesn't work correctly, will hang often, try it a few times:

# # Interrupt with CTRL+C while `something` is busy
# # Takes one KeyboardInterrupts to get stuck, and another to terminate fully
# with multiprocessing.Manager() as manager:
#     with ProcessPoolExecutor() as executor:
#         futures = [executor.submit(something)]
# 
#         for f in futures:
#             f.result()

I am not too familiar with the exact inner workings of these two context managers, but as a user, there was at least nothing to make me aware that the 2nd example is bad. If it's not a bug, and just incorrect ordering, perhaps ProcessPoolExecutor could raise an exception or print a warning that it shouldn't be used inside a Manager context in such a way.

CPython versions tested on:

3.9, 3.10, 3.11, 3.12

Operating systems tested on:

Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the two context-manager orderings with the provided ProcessPoolExecutor and multiprocessing.Manager example, then inspect multiprocessing.managers.py at serve_forever and the related shutdown paths. Done means the problematic ordering no longer hangs after KeyboardInterrupt and both orderings terminate consistently; no test file is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.