box / box/flaky

Flaky reruns tests with failing teardown but test suite is failed

Open
#159 3 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
398
Forks
57
PR merge metrics
No merged PRs in 30d

Description

`flaky` re-runs tests that fail in teardown, but test suite result is always `ERROR`. 🤔

Meanwhile if test fails in the test itself, it is reran, and test suite result is success. 👌

With this example, `flaky` does the right thing:

```py
# test_a.py
> cat test_a.py
import random
from flaky import flaky

@flaky(max_runs=3)
def test_foo():
assert random.random() < 0.3
```

result

```
> pytest test_a.py -v
========================== test session starts ==========================
platform darwin -- Python 3.7.3, pytest-4.6.5, py-1.8.0, pluggy-0.12.0 -- /.../bin/python
cachedir: .pytest_cache
rootdir: /...
plugins: asyncio-0.10.0, cov-2.7.1, flaky-3.6.1, freezegun-0.3.0.post1
collected 1 item

test_a.py::test_foo PASSED [100%]
===Flaky Test Report===

test_foo failed (2 runs remaining out of 3).

assert 0.6181257110745261 < 0.3
+ where 0.6181257110745261 = ()
+ where = random.random
[]
test_foo failed (1 runs remaining out of 3).

assert 0.6575059088357618 < 0.3
+ where 0.6575059088357618 = ()
+ where = random.random
[]
test_foo passed 1 out of the required 1 times. Success!

===End Flaky Test Report===

======================= 1 passed in 0.04 seconds ========================
```

And with this example, the result is weird:

```py
# test_b.py
import random
import pytest
from flaky import flaky

@pytest.fixture
def foobar():
# no setup
yield 42
assert random.random() < 0.3

@flaky(max_runs=3)
def test_foo(foobar):
pass
```

result

```
> pytest test_b.py
========================== test session starts ==========================
platform darwin -- Python 3.7.3, pytest-4.6.5, py-1.8.0, pluggy-0.12.0
rootdir: /...
plugins: asyncio-0.10.0, cov-2.7.1, flaky-3.6.1, freezegun-0.3.0.post1
collected 1 item

test_b.py .E [100%]

================================ ERRORS =================================
_____________________ ERROR at teardown of test_foo _____________________

@pytest.fixture
def foobar():
# no setup
yield 42
> assert random.random() < 0.3
E assert 0.7628703616086037 < 0.3
E + where 0.7628703616086037 = ()
E + where = random.random

test_b.py:10: AssertionError
===Flaky Test Report===

test_foo passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
=================== 1 passed, 1 error in 0.06 seconds ===================
```

Might be related to #135 but this is more specific and concise.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.