pytest-dev / pytest-dev/pytest

Test hangs when using mp.Pool on Windows/macOS using CLI runner when --import-mode=importlib

Open
#10,965 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

platform: mac platform: windows type: bug
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

This is a duplicate of #5429 (which does not appear to actually be resolved, or has regressed) but I come bearing more details, in particular a matrix that should help diagnose the source of the problem.

Here's a minimal example:

from multiprocessing import Pool

def times_two(x):
    return x * 2

def test_foo():
    with Pool(1) as pool:
        assert pool.map(times_two, [1, 2, 3, 4]) == [2, 4, 6, 8]

This is using pytest 7.3.1.

Tests were performed on Github Actions ubuntu-latest, windows-latest, and macos-latest, which correspond to Ubuntu 22.04, Windows Server 2022, and macOS 12 Monterey (not Ventura). I also validated the macOS and Linux results on macOS 13 Ventura and Ubuntu 20.04 on my own machines.

All of the below were tested with Python 3.11 on Github Actions runners. On my own machines, I tested with both 3.10 and 3.11, same results.

Platform Runner Result
Windows pytest Hangs
macOS pytest Hangs
Linux pytest Completes
macOS PyCharm Completes

The last row in particular suggests that the issue is likely in the test runner. Running all tests via PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_pytest_runner.py does not hang.

– Dylan

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the minimal multiprocessing example with the CLI runner on Windows and macOS using Python 3.11 and --import-mode=importlib, then compare it with Linux and the PyCharm runner. Investigate the test-runner path involved in collection or process startup; done means the example completes consistently on the affected platforms without regressing the Linux behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.