box / box/flaky

Output information during the test run

Open
#154 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
398
Forks
57
PR merge metrics
No merged PRs in 30d

Description

Depending on test runtime and repetition count flaky tests can take a long time to complete.

It would be nice if there was some feedback on flakyness during the test run, not only afterwards.

For now we're using the following as a slightly hacky workaround but a built in solution would be much nicer:
```python
@pytest.hookimpl(hookwrapper=True, trylast=True)
def pytest_runtest_call(item):
""" More feedback for flaky tests.

In verbose mode this outputs 'FLAKY' every time a test marked as flaky fails.
This doesn't work under xdist and will therefore show no output.
"""
outcome = yield
did_fail = isinstance(outcome._excinfo, tuple) and isinstance(
outcome._excinfo[1], BaseException
)
is_xdist = "PYTEST_XDIST_WORKER" in os.environ
is_flaky_test = item.get_closest_marker("flaky") is not None
if did_fail and is_flaky_test and not is_xdist:
if item.config.option.verbose > 0:
capmanager = item.config.pluginmanager.getplugin("capturemanager")
with capmanager.global_and_fixture_disabled():
item.config.pluginmanager.get_plugin("terminalreporter")._tw.write(
f"FLAKY ", yellow=True
)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.