pytest-dev / pytest-dev/pytest-rerunfailures
only_rerun with exception raised from fixture teardown reruns test but report previous runs as failure
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 477
- Forks
- 107
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 21
Description
Hi.
I have a need to do some checks in a fixture teardown to determine whether a test should rerun or not.
Because rerunfailures only allows filtering based on exception, what I had in mind is that to check whatever I want in the fixture and raise the custom exception so the test will rerun.
However, while this package detects a rerun is needed when processing the teardown report, the call (test) report was already processed and it was not marked to be rerun, because the rerun_only condition didn't match the test exception, as the custom exception was raised from the fixture.
Eventually, the test does re-run until it pass, but pytest still report the test failure outcomes.
Note that the reruns works just fine when not using only_rerun condition.
Example code:
import random
import pytest
class MyCustomError(Exception):
pass
@pytest.fixture
def rerun_custom_checker(request):
yield
if request.node.rep_call.outcome == "passed":
return
raise MyCustomError()
@pytest.mark.flaky(reruns=10, reruns_delay=1, only_rerun=["MyCustomError"])
def test_example(rerun_custom_checker):
i = random.randint(2, 4)
if i != 3:
assert False
What I had in mind, is to pre-process the reports and set a flag whether rerun is needed, if it was found to be needed for setup/call/teardown.
Then, when processing the reports as usual, and just check for this flag.
Something like:
I can raise a PR with that change, assuming this solution is acceptable (It works fine when testing this change)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the report-processing path for setup, call, and teardown, then trace how the rerun_only exception filter decides whether a test is rerun. Reproduce the fixture-teardown MyCustomError example and verify that teardown-triggered reruns are included in the final outcome rather than reported as failures from earlier runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100