pytest-dev / pytest-dev/pytest-xdist
looponfail does not re-run only the failed tests when distribution is enabled
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
looponfail is supposed to re-run only the tests that previously failed. It works per the expectation without distribution, but systematically re-runs all the tests (pass and fail) when distribution is enabled.
py.test test_looponfail.py --looponfail (OK)
py.test test_looponfail.py -n=1 --looponfail (NOK)
Please see all the logs below
I'm using:
python: 3.5.2
pytest: 3.1.2
pytest-xdist: 1.17.1
test_looponfail.py:
class TestLoopOnFail:
def test_this_test_should_pass(self):
time.sleep(1)
assert True, 'the test passed'
def test_this_test_should_fail(self):
time.sleep(2)
assert False, 'the test failed'
Case 1: Without distribution, only the test that fails re-runs
~/temp/pytest_examples/looponfail >py.test test_looponfail.py --looponfail
======================= test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/origaut/temp/pytest_examples/looponfail, inifile:
plugins: xdist-1.17.1
collected 2 items
collected 2 items
test_looponfail.py .F
======================= FAILURES ===================================
__________________ TestLoopOnFail.test_this_test_should_fail ___________________
self = <test_looponfail.TestLoopOnFail object at 0x102df1b00>
def test_this_test_should_fail(self):
time.sleep(1)
assert False, 'the test failed'
E AssertionError: the test failed
E assert False
test_looponfail.py:17: AssertionError
====================== 1 failed, 1 passed in 2.05 seconds ======================
############################ LOOPONFAILING ################################
test_looponfail.py::TestLoopOnFail::()::test_this_test_should_fail
############################ waiting for changes ###############################
### Watching: /Users/origaut/temp/pytest_examples/looponfail
# MODIFIED /Users/origaut/temp/pytest_examples/looponfail/test_looponfail.py
============================= test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/origaut/temp/pytest_examples/looponfail, inifile:
plugins: xdist-1.17.1
collected 3 items
collected 3 items
test_looponfail.py F
=================================== FAILURES ===================================
__________________ TestLoopOnFail.test_this_test_should_fail ___________________
self = <test_looponfail.TestLoopOnFail object at 0x1039e01d0>
def test_this_test_should_fail(self):
time.sleep(2)
assert False, 'the test failed'
E AssertionError: the test failed
E assert False
test_looponfail.py:17: AssertionError
=========================== 1 failed in 2.05 seconds ===========================
########################### LOOPONFAILING #################################
########################### waiting for changes ###############################
Case 2: With distribution, it re-runs all the tests
~/temp/pytest_examples/looponfail >py.test test_looponfail.py -n=1 --looponfail
============================ test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/origaut/temp/pytest_examples/looponfail, inifile:
plugins: xdist-1.17.1
gw0 I
gw0 [2]
scheduling tests via LoadScheduling
.F
================================== FAILURES ==============================
__________________ TestLoopOnFail.test_this_test_should_fail ___________________
[gw0] darwin -- Python 3.5.2 /Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5
self = <test_looponfail.TestLoopOnFail object at 0x1049f3160>
def test_this_test_should_fail(self):
time.sleep(2)
assert False, 'the test failed'
E AssertionError: the test failed
E assert False
test_looponfail.py:16: AssertionError
====================== 1 failed, 1 passed in 3.42 seconds ======================
####################### LOOPONFAILING #####################################
test_looponfail.py::TestLoopOnFail::()::test_this_test_should_fail
####################### waiting for changes ###################################
### Watching: /Users/origaut/temp/pytest_examples/looponfail
# MODIFIED /Users/origaut/temp/pytest_examples/looponfail/test_looponfail.py
=========================== test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/origaut/temp/pytest_examples/looponfail, inifile:
plugins: xdist-1.17.1
gw0 I
gw0 [2]
scheduling tests via LoadScheduling
.F
============================= FAILURES ===================================
__________________ TestLoopOnFail.test_this_test_should_fail ___________________
[gw0] darwin -- Python 3.5.2 /Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5
self = <test_looponfail.TestLoopOnFail object at 0x103fddfd0>
def test_this_test_should_fail(self):
time.sleep(1)
assert False, 'the test failed'
E AssertionError: the test failed
E assert False
test_looponfail.py:16: AssertionError
====================== 1 failed, 1 passed in 2.45 seconds ======================
####################### LOOPONFAILING #####################################
test_looponfail.py::TestLoopOnFail::()::test_this_test_should_fail
####################### waiting for changes ###################################
### Watching: /Users/origaut/temp/pytest_examples/looponfail
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with test_looponfail.py using py.test test_looponfail.py --looponfail and py.test test_looponfail.py -n=1 --looponfail. Compare the rerun selection after a file change; done means distribution with -n=1 reruns only the previously failing test, not the passing test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100