pytest-dev / pytest-dev/pytest-xdist

pytest-xdist causes warnings to be emitted when a unit test uses os.fork()

Open
#1,186 13 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

test suite:

import time
import multiprocessing

class TestWhatever:

    def test_thing(self):

        def go():
            time.sleep(2)

        ctx = multiprocessing.get_context("fork")
        proc = ctx.Process(target=go, args=())
        proc.start()

Running as pytest test.py -n1, output:

[classic@framework tmp2]$ pytest test.py -n1
========================================================================================== test session starts ==========================================================================================
platform linux -- Python 3.12.9, pytest-8.1.0, pluggy-1.4.0
rootdir: /home/classic/tmp2
plugins: xdist-3.4.0, anyio-4.1.0, random-0.2, repeat-0.9.3
1 worker [1 item]      
.                                                                                                                                                                                                 [100%]
=========================================================================================== warnings summary ============================================================================================
test.py::TestWhatever::test_thing
  /usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=16078) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 1 passed, 1 warning in 2.19s ======================================================================================

Per the author of this warning, multithreaded code is never safe if it also spawns using fork (see discussion). However I cannot locate any threads running. Here's an extension of the example that lists out threads running, and I can find none that are not the "main" thread:

conftest.py:

# conftest.py
import threading
import os
import logging

logging.basicConfig()
logging.getLogger("main").setLevel(logging.INFO)


class XDistHooks:
    def pytest_configure_node(self, node):
        for t in threading.enumerate():
            logging.getLogger("main").info(
                f"THREAD FROM MAIN PROCESS {os.getpid()}: {t}\n")


def pytest_configure(config):

    if config.pluginmanager.hasplugin("xdist"):
        config.pluginmanager.register(XDistHooks())


test.py:

# test.py
import time
import os
import multiprocessing
import logging
import threading

logging.basicConfig()
logging.getLogger("main").setLevel(logging.INFO)

class TestWhatever:

    def test_thing(self):
        for t in threading.enumerate():
            logging.getLogger("main").info(
                f"THREAD FROM CHILD PROCESS {os.getpid()} "
                f"(parent: {os.getppid()}): {t}\n")

        def go():
            time.sleep(10)

        ctx = multiprocessing.get_context("fork")
        proc = ctx.Process(target=go, args=())
        proc.start()

run output:

[classic@framework tmp]$ pytest test.py   -s -p no:logging -n1
========================================================================================== test session starts ==========================================================================================
platform linux -- Python 3.12.9, pytest-8.1.0, pluggy-1.4.0
rootdir: /home/classic/tmp
plugins: xdist-3.4.0, anyio-4.1.0, random-0.2, repeat-0.9.3
initialized: 1/1 workerINFO:main:THREAD FROM MAIN PROCESS 16341: <_MainThread(MainThread, started 139752741145472)>

1 worker [1 item]      
INFO:main:THREAD FROM CHILD PROCESS 16342 (parent: 16341): <_MainThread(MainThread, started 139984508341120)>

.
=========================================================================================== warnings summary ============================================================================================
test.py::TestWhatever::test_thing
  /usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=16342) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 1 passed, 1 warning in 10.17s =====================================================================================

Basically I want to keep using fork() in my test code, since we are running functions inside the tests themselves in processes. Where is pytest-xdist and/or execnet spawning threads exactly (code is pretty opaque) and is this a bug in the python interpreter?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported test.py and conftest.py reproducer and run pytest test.py -n1 with the thread logging enabled. Read multiprocessing/popen_fork.py:66, then trace pytest-xdist and execnet startup to identify the source of any threads. Done means establishing whether the warning is caused by xdist/execnet or Python itself and documenting an actionable fix or explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.