pytest-dev / pytest-dev/pytest-xdist

`--maxfail` does not work correctly

Open
#868 5 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

We noticed an issue with pytest-xdist in combination with --maxfail>0.
It is related to https://github.com/pytest-dev/pytest-xdist/issues/420, but goes even further.
For some reason, tearDownClass is not executed completely at the end of a failed test.

This is an example TestCase where the issue occurs:

class TestCase(unittest.TestCase):
    def setUp(self) -> None:
        logging.info("setUp started")
        for _ in range(4):
            time.sleep(3)
            logging.info("setup running")
        logging.info("setUp executed")
        raise TimeoutError

    def test_1(self):
        pass

    def test_2(self):
        pass

    @classmethod
    def tearDownClass(cls) -> None:
        logging.info("tearDownClass started")
        time.sleep(10)
        logging.info("tearDownClass executed")
Results without pytest-xdist: pytest --maxfail=1 -s --log-cli-level=INFO test_test.py 
================================= test session starts ==================================
platform darwin -- Python 3.7.14, pytest-7.2.0, pluggy-0.13.1
rootdir: /Users/d065057/ML-MEH-Assembly/dar_end_to_end_tests
plugins: xdist-3.1.0, timeout-2.1.0
collected 2 items                                                                      

test_test.py::TestCase::test_1 
------------------------------------ live log call -------------------------------------
INFO     root:test_test.py:8 setUp started
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:12 setUp executed
FAILED
test_test.py::TestCase::test_2 
------------------------------------ live log call -------------------------------------
INFO     root:test_test.py:8 setUp started
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:12 setUp executed
FAILED
---------------------------------- live log teardown -----------------------------------
INFO     root:test_test.py:23 tearDownClass started
INFO     root:test_test.py:25 tearDownClass executed


======================================= FAILURES =======================================
___________________________________ TestCase.test_1 ____________________________________

self = <dar_end_to_end_tests.test_test.TestCase testMethod=test_1>

    def setUp(self) -> None:
        logging.info("setUp started")
        for _ in range(4):
            time.sleep(3)
            logging.info("setup running")
        logging.info("setUp executed")
>       raise TimeoutError
E       TimeoutError

test_test.py:13: TimeoutError
---------------------------------- Captured log call -----------------------------------
INFO     root:test_test.py:8 setUp started
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:12 setUp executed
___________________________________ TestCase.test_2 ____________________________________

self = <dar_end_to_end_tests.test_test.TestCase testMethod=test_2>

    def setUp(self) -> None:
        logging.info("setUp started")
        for _ in range(4):
            time.sleep(3)
            logging.info("setup running")
        logging.info("setUp executed")
>       raise TimeoutError
E       TimeoutError

test_test.py:13: TimeoutError
---------------------------------- Captured log call -----------------------------------
INFO     root:test_test.py:8 setUp started
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:12 setUp executed
-------------------------------- Captured log teardown ---------------------------------
INFO     root:test_test.py:23 tearDownClass started
INFO     root:test_test.py:25 tearDownClass executed
=============================== short test summary info ================================
FAILED test_test.py::TestCase::test_1 - TimeoutError
FAILED test_test.py::TestCase::test_2 - TimeoutError
================================== 2 failed in 34.12s ==================================
 ~/ML-MEH-Assembly/dar_end_to_end_tests  pytest_xdist_poc *1 +3 !2 ?2  pytest --maxfail=1 -s --log-cli-level=INFO test_test.py
================================= test session starts ==================================
platform darwin -- Python 3.7.14, pytest-7.2.0, pluggy-0.13.1
rootdir: /Users/d065057/ML-MEH-Assembly/dar_end_to_end_tests
plugins: xdist-3.1.0, timeout-2.1.0
collected 2 items                                                                      

test_test.py::TestCase::test_1 
------------------------------------ live log call -------------------------------------
INFO     root:test_test.py:8 setUp started
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:12 setUp executed
FAILED-------------------------------- live log sessionfinish --------------------------------
INFO     root:test_test.py:23 tearDownClass started
INFO     root:test_test.py:25 tearDownClass executed


======================================= FAILURES =======================================
___________________________________ TestCase.test_1 ____________________________________

self = <dar_end_to_end_tests.test_test.TestCase testMethod=test_1>

    def setUp(self) -> None:
        logging.info("setUp started")
        for _ in range(4):
            time.sleep(3)
            logging.info("setup running")
        logging.info("setUp executed")
>       raise TimeoutError
E       TimeoutError

test_test.py:13: TimeoutError
---------------------------------- Captured log call -----------------------------------
INFO     root:test_test.py:8 setUp started
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:12 setUp executed
=============================== short test summary info ================================
FAILED test_test.py::TestCase::test_1 - TimeoutError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================== 1 failed in 22.08s ==================================
Results with pytest-xdist: pytest -n 1 --dist loadscope --maxfail=1 -s --log-cli-level=INFO test_test.py

(Executing with this trick to make logs visible)

================================= test session starts ==================================
platform darwin -- Python 3.7.14, pytest-7.2.0, pluggy-0.13.1
rootdir: /Users/d065057/ML-MEH-Assembly/dar_end_to_end_tests
plugins: xdist-3.1.0, timeout-2.1.0
gw0 C/Users/d065057/ML-MEH-Assembly/dar_end_to_end_tests/conftest.py:5: ResourceWarning: unclosed file <_io.TextIOWrapper name=1 mode='w' encoding='UTF-8'>
  sys.stdout = sys.stderr
ResourceWarning: Enable tracemalloc to get the object allocation traceback
[gw0] Python 3.7.14 (default, Sep  6 2022, 23:37:35)  -- [Clang 13.1.6 (clang-1316.0.21.2.5)]
gw0 ok================================= test session starts ==================================
platform darwin -- Python 3.7.14, pytest-7.2.0, pluggy-0.13.1
rootdir: /Users/d065057/ML-MEH-Assembly/dar_end_to_end_tests
plugins: xdist-3.1.0, timeout-2.1.0
collected 2 items                                                                      
gw0 [2]
scheduling tests via LoadScopeScheduling

test_test.py::TestCase::test_1 
test_test.py::TestCase::test_1 
------------------------------------ live log call -------------------------------------
INFO     root:test_test.py:8 setUp started
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:12 setUp executed
FAILED
[gw0] FAILED test_test.py::TestCase::test_1 
test_test.py::TestCase::test_2 
------------------------------------ live log call -------------------------------------
INFO     root:test_test.py:8 setUp started
INFO     root:test_test.py:11 setup running
-------------------------------- live log sessionfinish --------------------------------
INFO     root:test_test.py:23 tearDownClass started


======================================= FAILURES =======================================
___________________________________ TestCase.test_1 ____________________________________
[gw0] darwin -- Python 3.7.14 /Users/d065057/ML-MEH-Assembly/venv/bin/python

self = <dar_end_to_end_tests.test_test.TestCase testMethod=test_1>

    def setUp(self) -> None:
        logging.info("setUp started")
        for _ in range(4):
            time.sleep(3)
            logging.info("setup running")
        logging.info("setUp executed")
>       raise TimeoutError
E       TimeoutError

test_test.py:13: TimeoutError
---------------------------------- Captured log call -----------------------------------
INFO     root:test_test.py:8 setUp started
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:11 setup running
INFO     root:test_test.py:12 setUp executed
=============================== short test summary info ================================
FAILED test_test.py::TestCase::test_1 - TimeoutError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!
================================== 1 failed in 22.42s ==================================

As you can see in the logs, pytest usually (without pytest-xdist) executes setUp only once and then executes tearDownClass until its completion in the end.

Contrary to this, with pytest-xdist, setUp is executed twice: The first time until it is finished, and the second time it is aborted. Moreover, tearDownClass is started, but aborted.

Is this expected behavior? If yes, is this behavior documented somewhere? Do you have any recommendations on how to fix the issue?

The most important point for us would be to always execute tearDownClass fully in the end.
(It would also be good if setUp would only be executed once).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the behavior from the test_test.py example using pytest -n 1 --dist loadscope --maxfail=1, comparing it with the non-xdist command. Inspect conftest.py and the pytest-xdist scheduling and interruption behavior; done means the intended teardown and setup behavior is established and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.