pytest-dev / pytest-dev/pytest-xdist

looponfail does not re-run only the failed tests when distribution is enabled

Open
#165 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

looponfail is supposed to re-run only the tests that previously failed. It works per the expectation without distribution, but systematically re-runs all the tests (pass and fail) when distribution is enabled.

py.test test_looponfail.py --looponfail (OK)
py.test test_looponfail.py -n=1 --looponfail (NOK)

Please see all the logs below

I'm using:
python: 3.5.2
pytest: 3.1.2
pytest-xdist: 1.17.1

test_looponfail.py:

class TestLoopOnFail:

def test_this_test_should_pass(self):
    time.sleep(1)
    assert True, 'the test passed'

def test_this_test_should_fail(self):
    time.sleep(2)
    assert False, 'the test failed'

Case 1: Without distribution, only the test that fails re-runs

~/temp/pytest_examples/looponfail >py.test test_looponfail.py --looponfail
======================= test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/origaut/temp/pytest_examples/looponfail, inifile:
plugins: xdist-1.17.1
collected 2 items
collected 2 items

test_looponfail.py .F

======================= FAILURES ===================================
__________________ TestLoopOnFail.test_this_test_should_fail ___________________

self = <test_looponfail.TestLoopOnFail object at 0x102df1b00>

def test_this_test_should_fail(self):
    time.sleep(1)
  assert False, 'the test failed'

E AssertionError: the test failed
E assert False

test_looponfail.py:17: AssertionError
====================== 1 failed, 1 passed in 2.05 seconds ======================
############################ LOOPONFAILING ################################
test_looponfail.py::TestLoopOnFail::()::test_this_test_should_fail
############################ waiting for changes ###############################
### Watching: /Users/origaut/temp/pytest_examples/looponfail
# MODIFIED /Users/origaut/temp/pytest_examples/looponfail/test_looponfail.py
============================= test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/origaut/temp/pytest_examples/looponfail, inifile:
plugins: xdist-1.17.1
collected 3 items
collected 3 items

test_looponfail.py F

=================================== FAILURES ===================================
__________________ TestLoopOnFail.test_this_test_should_fail ___________________

self = <test_looponfail.TestLoopOnFail object at 0x1039e01d0>

def test_this_test_should_fail(self):
    time.sleep(2)
  assert False, 'the test failed'

E AssertionError: the test failed
E assert False

test_looponfail.py:17: AssertionError
=========================== 1 failed in 2.05 seconds ===========================
########################### LOOPONFAILING #################################
########################### waiting for changes ###############################

Case 2: With distribution, it re-runs all the tests

~/temp/pytest_examples/looponfail >py.test test_looponfail.py -n=1 --looponfail
============================ test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/origaut/temp/pytest_examples/looponfail, inifile:
plugins: xdist-1.17.1
gw0 I
gw0 [2]

scheduling tests via LoadScheduling
.F
================================== FAILURES ==============================
__________________ TestLoopOnFail.test_this_test_should_fail ___________________
[gw0] darwin -- Python 3.5.2 /Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5
self = <test_looponfail.TestLoopOnFail object at 0x1049f3160>

def test_this_test_should_fail(self):
    time.sleep(2)
  assert False, 'the test failed'

E AssertionError: the test failed
E assert False

test_looponfail.py:16: AssertionError
====================== 1 failed, 1 passed in 3.42 seconds ======================
####################### LOOPONFAILING #####################################
test_looponfail.py::TestLoopOnFail::()::test_this_test_should_fail
####################### waiting for changes ###################################
### Watching: /Users/origaut/temp/pytest_examples/looponfail
# MODIFIED /Users/origaut/temp/pytest_examples/looponfail/test_looponfail.py
=========================== test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/origaut/temp/pytest_examples/looponfail, inifile:
plugins: xdist-1.17.1
gw0 I
gw0 [2]

scheduling tests via LoadScheduling
.F
============================= FAILURES ===================================
__________________ TestLoopOnFail.test_this_test_should_fail ___________________
[gw0] darwin -- Python 3.5.2 /Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5
self = <test_looponfail.TestLoopOnFail object at 0x103fddfd0>

def test_this_test_should_fail(self):
    time.sleep(1)
  assert False, 'the test failed'

E AssertionError: the test failed
E assert False

test_looponfail.py:16: AssertionError
====================== 1 failed, 1 passed in 2.45 seconds ======================
####################### LOOPONFAILING #####################################
test_looponfail.py::TestLoopOnFail::()::test_this_test_should_fail
####################### waiting for changes ###################################
### Watching: /Users/origaut/temp/pytest_examples/looponfail

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with test_looponfail.py using py.test test_looponfail.py --looponfail and py.test test_looponfail.py -n=1 --looponfail. Compare the rerun selection after a file change; done means distribution with -n=1 reruns only the previously failing test, not the passing test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.