pytest-dev / pytest-dev/pytest

ModuleNotFoundError with multiprocessing and import-mode=importlib

Open
#12,178 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

topic: collection
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

In https://github.com/jaraco/keyring/issues/673, I captured an issue stemming from the introduction of --import-mode importlib to jaraco/skeleton projects. In https://github.com/jaraco/keyring/commit/6ff02e0eefcd90e271cefd326b460ecfa0e3eb9e, I'm attempting to enable importlib mode for all projects in order to support projects that use namespaces and those that don't in a uniform way. However, enabling importlib mode causes failures in tests that involve multiprocessing. Consider this minimal example:

 @ cat > tox.ini
[testenv]
deps =
	pytest
commands =
	pytest {posargs:--import-mode=importlib}
 @ mkdir tests
 @ cat > tests/test_multiprocess.py
import multiprocessing


def multiprocess_do():
    pass


def test_multiprocess():
    proc1 = multiprocessing.Process(target=multiprocess_do)
    proc1.start()
    proc1.join()
    assert proc1.exitcode == 0
 @ tree
.
├── tests
│   └── test_multiprocess.py
└── tox.ini

2 directories, 2 files
 @ tox
py: commands[0]> pytest --import-mode=importlib
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
cachedir: .tox/py/.pytest_cache
rootdir: /Users/jaraco/draft
collected 1 item

tests/test_multiprocess.py F                                             [100%]

=================================== FAILURES ===================================
______________________________ test_multiprocess _______________________________

    def test_multiprocess():
        proc1 = multiprocessing.Process(target=multiprocess_do)
        proc1.start()
        proc1.join()
>       assert proc1.exitcode == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <Process name='Process-1' pid=38572 parent=38570 stopped exitcode=1>.exitcode

tests/test_multiprocess.py:12: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'tests'
=========================== short test summary info ============================
FAILED tests/test_multiprocess.py::test_multiprocess - AssertionError: assert...
============================== 1 failed in 0.09s ===============================
py: exit 1 (0.19 seconds) /Users/jaraco/draft> pytest --import-mode=importlib pid=38570
  py: FAIL code 1 (0.22=setup[0.02]+cmd[0.19] seconds)
  evaluation failed :( (0.25 seconds)

The same error occurs even if tests/__init__.py is present. The issue occurs also with pytest 8.0.2 and 7.4.4. My guess is that it never worked with this mode.

It seems that tests.test_multiprocess is in sys.modules and objects passed between the processes might reference that module, so the child process needs to have the same path handling that the parent had but doesn't.

Running tox -- --import-mode prepend or tox -- --import-mode append bypasses the failure.

Can multiprocessing be made to work with import-mode importlib?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal reproduction in tests/test_multiprocess.py and the pytest command in tox.ini, then run pytest --import-mode=importlib to observe the child-process failure. Compare it with prepend and append modes. Done means the multiprocessing test exits successfully under importlib mode without breaking the existing import modes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.