box / box/flaky

pytest unittest cases are destroyed after first teardown

Open
#212 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
398
Forks
57
PR merge metrics
No merged PRs in 30d

Description

The following test from `test/test_pytest/test_pytest_example.py` fails:

```python

@flaky
class TestExampleFlakyTestCase(TestCase):
_threshold = -1

@staticmethod
def test_flaky_thing_that_fails_then_succeeds():
"""
Flaky will run this test twice.
It will fail once and then succeed.
"""
TestExampleFlakyTestCase._threshold += 1
assert TestExampleFlakyTestCase._threshold >= 1

```
```
========================================================================================================================== FAILURES ==========================================================================================================================
_____________________________________________________________________________________________ TestExampleFlakyTestCase.test_flaky_thing_that_fails_then_succeeds _____________________________________________________________________________________________

> lambda: ihook(item=item, **kwds), when=when, reraise=reraise
)
E AssertionError

flaky/flaky_pytest_plugin.py:148: AssertionError
===Flaky Test Report===

test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2).

assert 0 >= 1
+ where 0 = TestExampleFlakyTestCase._threshold
[, , , /python-flaky-3.8.1/test/test_pytest/test_pytest_example.py:101>]
test_flaky_thing_that_fails_then_succeeds failed; it passed 0 out of the required 1 times.

[/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/runner.py:342>, /python-flaky-3.8.1/flaky/flaky_pytest_plugin.py:148>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_hooks.py:513>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_manager.py:120>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:139>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/threadexception.py:87>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/threadexception.py:63>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:90>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:65>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/logging.py:850>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/logging.py:833>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/capture.py:878>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/skipping.py:257>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:103>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/runner.py:184>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/runner.py:174>, /python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/unittest.py:319>]

===End Flaky Test Report===
================================================================================================================== short test summary info ===================================================================================================================
FAILED test/test_pytest/test_pytest_example.py::TestExampleFlakyTestCase::test_flaky_thing_that_fails_then_succeeds - AssertionError
===================================================================================================================== 1 failed in 0.04s ======================================================================================================================
```
The first iteration fails (expected) and then each subsequent iteration fails due to an assertion error in pytest:

```
def runtest(self) -> None:
from _pytest.debugging import maybe_wrap_pytest_function_for_tracing

testcase = self.instance
> assert testcase is not None
E AssertionError

```

This appears to be due to a bug in pytest itself https://github.com/pytest-dev/pytest/issues/12424

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.