pytest-dev / pytest-dev/pytest
ModuleNotFoundError with multiprocessing and import-mode=importlib
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
In https://github.com/jaraco/keyring/issues/673, I captured an issue stemming from the introduction of --import-mode importlib to jaraco/skeleton projects. In https://github.com/jaraco/keyring/commit/6ff02e0eefcd90e271cefd326b460ecfa0e3eb9e, I'm attempting to enable importlib mode for all projects in order to support projects that use namespaces and those that don't in a uniform way. However, enabling importlib mode causes failures in tests that involve multiprocessing. Consider this minimal example:
@ cat > tox.ini
[testenv]
deps =
pytest
commands =
pytest {posargs:--import-mode=importlib}
@ mkdir tests
@ cat > tests/test_multiprocess.py
import multiprocessing
def multiprocess_do():
pass
def test_multiprocess():
proc1 = multiprocessing.Process(target=multiprocess_do)
proc1.start()
proc1.join()
assert proc1.exitcode == 0
@ tree
.
├── tests
│ └── test_multiprocess.py
└── tox.ini
2 directories, 2 files
@ tox
py: commands[0]> pytest --import-mode=importlib
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
cachedir: .tox/py/.pytest_cache
rootdir: /Users/jaraco/draft
collected 1 item
tests/test_multiprocess.py F [100%]
=================================== FAILURES ===================================
______________________________ test_multiprocess _______________________________
def test_multiprocess():
proc1 = multiprocessing.Process(target=multiprocess_do)
proc1.start()
proc1.join()
> assert proc1.exitcode == 0
E AssertionError: assert 1 == 0
E + where 1 = <Process name='Process-1' pid=38572 parent=38570 stopped exitcode=1>.exitcode
tests/test_multiprocess.py:12: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
self = reduction.pickle.load(from_parent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'tests'
=========================== short test summary info ============================
FAILED tests/test_multiprocess.py::test_multiprocess - AssertionError: assert...
============================== 1 failed in 0.09s ===============================
py: exit 1 (0.19 seconds) /Users/jaraco/draft> pytest --import-mode=importlib pid=38570
py: FAIL code 1 (0.22=setup[0.02]+cmd[0.19] seconds)
evaluation failed :( (0.25 seconds)
The same error occurs even if tests/__init__.py is present. The issue occurs also with pytest 8.0.2 and 7.4.4. My guess is that it never worked with this mode.
It seems that tests.test_multiprocess is in sys.modules and objects passed between the processes might reference that module, so the child process needs to have the same path handling that the parent had but doesn't.
Running tox -- --import-mode prepend or tox -- --import-mode append bypasses the failure.
Can multiprocessing be made to work with import-mode importlib?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the minimal reproduction in tests/test_multiprocess.py and the pytest command in tox.ini, then run pytest --import-mode=importlib to observe the child-process failure. Compare it with prepend and append modes. Done means the multiprocessing test exits successfully under importlib mode without breaking the existing import modes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100