pytest-dev / pytest-dev/pytest-xdist
stepwise doesn't stop the suite on first failure with xdist
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
Stepwise is supposed to behave like -x but this doesn't work with xdist:
You may think of it as a combination of the -x option, which exits the test run after a failing test, and the --lf option from pytest-cache, which only runs failing tests.
graingert@onomastic:~/projects/foo$ cat tests/test_foo.py
import pytest
def test_1():
assert True
def test_2():
assert True
def test_3():
assert False
def test_4():
assert True
def test_5():
assert True
def test_6():
assert True
Running it the first time, the test suite doesn't stop: (no -x behavior)
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [6]
scheduling tests via LoadScheduling
tests/test_foo.py::test_1
[gw0] [ 16%] PASSED tests/test_foo.py::test_1
tests/test_foo.py::test_2
[gw0] [ 33%] PASSED tests/test_foo.py::test_2
tests/test_foo.py::test_3
[gw0] [ 50%] FAILED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 66%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 83%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7
def test_3():
> assert False
E assert False
tests/test_foo.py:10: AssertionError
==================================================================================== 1 failed, 5 passed in 0.23 seconds ====================================================================================
Running it the second time, the test suite starts from the first failing test
but doesn't stop: (no -x behavior)
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [4]
scheduling tests via LoadScheduling
tests/test_foo.py::test_3
[gw0] [ 25%] FAILED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 50%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 75%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7
def test_3():
> assert False
E assert False
tests/test_foo.py:10: AssertionError
==================================================================================== 1 failed, 3 passed in 0.21 seconds ====================================================================================
Ok lets make the test pass:
graingert@onomastic:~/projects/foo$ cat tests/test_foo.py
import pytest
def test_1():
assert True
def test_2():
assert True
def test_3():
assert True
def test_4():
assert True
def test_5():
assert True
def test_6():
assert True
ok it starts from the failure and continues the suite, this is correct
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [4]
scheduling tests via LoadScheduling
tests/test_foo.py::test_3
[gw0] [ 25%] PASSED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 50%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 75%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
========================================================================================= 4 passed in 0.22 seconds =========================================================================================
and when run again the whole suite passes, this is correct too
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [6]
scheduling tests via LoadScheduling
tests/test_foo.py::test_1
[gw0] [ 16%] PASSED tests/test_foo.py::test_1
tests/test_foo.py::test_2
[gw0] [ 33%] PASSED tests/test_foo.py::test_2
tests/test_foo.py::test_3
[gw0] [ 50%] PASSED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 66%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 83%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
========================================================================================= 6 passed in 0.21 seconds =========================================================================================
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported commands with pytest, --sw, -n 1, and the example tests/test_foo.py suite. Compare the observed run with the expected -x behavior: execution should stop at the first failure while retaining stepwise restart behavior, and the suite should continue normally after that failure passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100