pytest-dev / pytest-dev/pytest-xdist

stepwise doesn't stop the suite on first failure with xdist

Open
#458 2 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

Stepwise is supposed to behave like -x but this doesn't work with xdist:

You may think of it as a combination of the -x option, which exits the test run after a failing test, and the --lf option from pytest-cache, which only runs failing tests.

graingert@onomastic:~/projects/foo$ cat tests/test_foo.py 
import pytest

def test_1():
    assert True

def test_2():
    assert True

def test_3():
    assert False

def test_4():
    assert True

def test_5():
    assert True

def test_6():
    assert True

Running it the first time, the test suite doesn't stop: (no -x behavior)

graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr  3 2019, 05:39:12)  -- [GCC 8.3.0]
gw0 [6]
scheduling tests via LoadScheduling

tests/test_foo.py::test_1 
[gw0] [ 16%] PASSED tests/test_foo.py::test_1 
tests/test_foo.py::test_2 
[gw0] [ 33%] PASSED tests/test_foo.py::test_2 
tests/test_foo.py::test_3 
[gw0] [ 50%] FAILED tests/test_foo.py::test_3 
tests/test_foo.py::test_4 
[gw0] [ 66%] PASSED tests/test_foo.py::test_4 
tests/test_foo.py::test_5 
[gw0] [ 83%] PASSED tests/test_foo.py::test_5 
tests/test_foo.py::test_6 
[gw0] [100%] PASSED tests/test_foo.py::test_6 

================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7

    def test_3():
>       assert False
E       assert False

tests/test_foo.py:10: AssertionError
==================================================================================== 1 failed, 5 passed in 0.23 seconds ====================================================================================

Running it the second time, the test suite starts from the first failing test
but doesn't stop: (no -x behavior)

graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr  3 2019, 05:39:12)  -- [GCC 8.3.0]
gw0 [4]
scheduling tests via LoadScheduling

tests/test_foo.py::test_3 
[gw0] [ 25%] FAILED tests/test_foo.py::test_3 
tests/test_foo.py::test_4 
[gw0] [ 50%] PASSED tests/test_foo.py::test_4 
tests/test_foo.py::test_5 
[gw0] [ 75%] PASSED tests/test_foo.py::test_5 
tests/test_foo.py::test_6 
[gw0] [100%] PASSED tests/test_foo.py::test_6 

================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7

    def test_3():
>       assert False
E       assert False

tests/test_foo.py:10: AssertionError
==================================================================================== 1 failed, 3 passed in 0.21 seconds ====================================================================================

Ok lets make the test pass:

graingert@onomastic:~/projects/foo$ cat tests/test_foo.py 
import pytest

def test_1():
    assert True

def test_2():
    assert True

def test_3():
    assert True

def test_4():
    assert True

def test_5():
    assert True

def test_6():
    assert True

ok it starts from the failure and continues the suite, this is correct

graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr  3 2019, 05:39:12)  -- [GCC 8.3.0]
gw0 [4]
scheduling tests via LoadScheduling

tests/test_foo.py::test_3 
[gw0] [ 25%] PASSED tests/test_foo.py::test_3 
tests/test_foo.py::test_4 
[gw0] [ 50%] PASSED tests/test_foo.py::test_4 
tests/test_foo.py::test_5 
[gw0] [ 75%] PASSED tests/test_foo.py::test_5 
tests/test_foo.py::test_6 
[gw0] [100%] PASSED tests/test_foo.py::test_6 

========================================================================================= 4 passed in 0.22 seconds =========================================================================================

and when run again the whole suite passes, this is correct too

graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr  3 2019, 05:39:12)  -- [GCC 8.3.0]
gw0 [6]
scheduling tests via LoadScheduling

tests/test_foo.py::test_1 
[gw0] [ 16%] PASSED tests/test_foo.py::test_1 
tests/test_foo.py::test_2 
[gw0] [ 33%] PASSED tests/test_foo.py::test_2 
tests/test_foo.py::test_3 
[gw0] [ 50%] PASSED tests/test_foo.py::test_3 
tests/test_foo.py::test_4 
[gw0] [ 66%] PASSED tests/test_foo.py::test_4 
tests/test_foo.py::test_5 
[gw0] [ 83%] PASSED tests/test_foo.py::test_5 
tests/test_foo.py::test_6 
[gw0] [100%] PASSED tests/test_foo.py::test_6 

========================================================================================= 6 passed in 0.21 seconds =========================================================================================

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported commands with pytest, --sw, -n 1, and the example tests/test_foo.py suite. Compare the observed run with the expected -x behavior: execution should stop at the first failure while retaining stepwise restart behavior, and the suite should continue normally after that failure passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.