pytest-dev / pytest-dev/pytest-xdist
pytest-xdist causes warnings to be emitted when a unit test uses os.fork()
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
test suite:
import time
import multiprocessing
class TestWhatever:
def test_thing(self):
def go():
time.sleep(2)
ctx = multiprocessing.get_context("fork")
proc = ctx.Process(target=go, args=())
proc.start()
Running as pytest test.py -n1, output:
[classic@framework tmp2]$ pytest test.py -n1
========================================================================================== test session starts ==========================================================================================
platform linux -- Python 3.12.9, pytest-8.1.0, pluggy-1.4.0
rootdir: /home/classic/tmp2
plugins: xdist-3.4.0, anyio-4.1.0, random-0.2, repeat-0.9.3
1 worker [1 item]
. [100%]
=========================================================================================== warnings summary ============================================================================================
test.py::TestWhatever::test_thing
/usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=16078) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 1 passed, 1 warning in 2.19s ======================================================================================
Per the author of this warning, multithreaded code is never safe if it also spawns using fork (see discussion). However I cannot locate any threads running. Here's an extension of the example that lists out threads running, and I can find none that are not the "main" thread:
conftest.py:
# conftest.py
import threading
import os
import logging
logging.basicConfig()
logging.getLogger("main").setLevel(logging.INFO)
class XDistHooks:
def pytest_configure_node(self, node):
for t in threading.enumerate():
logging.getLogger("main").info(
f"THREAD FROM MAIN PROCESS {os.getpid()}: {t}\n")
def pytest_configure(config):
if config.pluginmanager.hasplugin("xdist"):
config.pluginmanager.register(XDistHooks())
test.py:
# test.py
import time
import os
import multiprocessing
import logging
import threading
logging.basicConfig()
logging.getLogger("main").setLevel(logging.INFO)
class TestWhatever:
def test_thing(self):
for t in threading.enumerate():
logging.getLogger("main").info(
f"THREAD FROM CHILD PROCESS {os.getpid()} "
f"(parent: {os.getppid()}): {t}\n")
def go():
time.sleep(10)
ctx = multiprocessing.get_context("fork")
proc = ctx.Process(target=go, args=())
proc.start()
run output:
[classic@framework tmp]$ pytest test.py -s -p no:logging -n1
========================================================================================== test session starts ==========================================================================================
platform linux -- Python 3.12.9, pytest-8.1.0, pluggy-1.4.0
rootdir: /home/classic/tmp
plugins: xdist-3.4.0, anyio-4.1.0, random-0.2, repeat-0.9.3
initialized: 1/1 workerINFO:main:THREAD FROM MAIN PROCESS 16341: <_MainThread(MainThread, started 139752741145472)>
1 worker [1 item]
INFO:main:THREAD FROM CHILD PROCESS 16342 (parent: 16341): <_MainThread(MainThread, started 139984508341120)>
.
=========================================================================================== warnings summary ============================================================================================
test.py::TestWhatever::test_thing
/usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=16342) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 1 passed, 1 warning in 10.17s =====================================================================================
Basically I want to keep using fork() in my test code, since we are running functions inside the tests themselves in processes. Where is pytest-xdist and/or execnet spawning threads exactly (code is pretty opaque) and is this a bug in the python interpreter?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported test.py and conftest.py reproducer and run pytest test.py -n1 with the thread logging enabled. Read multiprocessing/popen_fork.py:66, then trace pytest-xdist and execnet startup to identify the source of any threads. Done means establishing whether the warning is caused by xdist/execnet or Python itself and documenting an actionable fix or explanation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100