box / box/flaky

Flaky re-runs xfail tests

Open
#184 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
398
Forks
57
PR merge metrics
No merged PRs in 30d

Description

If I mark a test as xfail (expected to fail), then flaky shouldn't re-run the test when it fails. Additionally, xfail tests are counted twice, while regular failing tests are counted once.

test_it.py:
```
import pytest

@pytest.mark.xfail
def test_xfail():
assert False

def test_fail():
assert False
```

Run with pytest:
```
% pytest -vv --force-flaky
============================================ test session starts ============================================
platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/virtualenvs/tmp-d591713aeced424/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/flakybug
plugins: flaky-3.7.0
collected 2 items

test_it.py::test_xfail XFAIL [ 50%]
test_it.py::test_xfail XFAIL [ 50%]
test_it.py::test_fail FAILED [100%]

================================================= FAILURES ==================================================
_________________________________________________ test_fail _________________________________________________

def test_fail():
> assert False
E assert False

test_it.py:8: AssertionError
===Flaky Test Report===

test_xfail failed (1 runs remaining out of 2).

assert False
[]
test_xfail failed; it passed 0 out of the required 1 times.

assert False
[]
test_fail failed (1 runs remaining out of 2).

assert False
[]
test_fail failed; it passed 0 out of the required 1 times.

assert False
[]

===End Flaky Test Report===
========================================== short test summary info ==========================================
FAILED test_it.py::test_fail - assert False
======================================= 1 failed, 2 xfailed in 0.06s ========================================
```

The xfail test is listed twice in the -vv lines, and counted twice in the summary at the end.

```
% pip freeze
attrs==21.2.0
flaky==3.7.0
iniconfig==1.1.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.6
pytest==6.2.5
toml==0.10.2
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the report with the test_it.py example and pytest --force-flaky, comparing the xfail and regular failing test output. Trace the flaky plugin's pytest handling; done means xfail tests are not re-run and appear once in verbose output, reports, and the final summary.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.