pytest-dev / pytest-dev/pytest-rerunfailures

only_rerun with exception raised from fixture teardown reruns test but report previous runs as failure

Open
#261 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
477
Forks
107
Avg merge
2d 14h
Merged PRs (30d)
21

Description

Hi.
I have a need to do some checks in a fixture teardown to determine whether a test should rerun or not.
Because rerunfailures only allows filtering based on exception, what I had in mind is that to check whatever I want in the fixture and raise the custom exception so the test will rerun.
However, while this package detects a rerun is needed when processing the teardown report, the call (test) report was already processed and it was not marked to be rerun, because the rerun_only condition didn't match the test exception, as the custom exception was raised from the fixture.

Eventually, the test does re-run until it pass, but pytest still report the test failure outcomes.

Note that the reruns works just fine when not using only_rerun condition.

Example code:

import random

import pytest

class MyCustomError(Exception):
    pass

@pytest.fixture
def rerun_custom_checker(request):
    yield

    if request.node.rep_call.outcome == "passed":
        return

    raise MyCustomError()


@pytest.mark.flaky(reruns=10, reruns_delay=1, only_rerun=["MyCustomError"])
def test_example(rerun_custom_checker):

    i = random.randint(2, 4)
    if i != 3:
        assert False
image

What I had in mind, is to pre-process the reports and set a flag whether rerun is needed, if it was found to be needed for setup/call/teardown.
Then, when processing the reports as usual, and just check for this flag.

Something like:
image

I can raise a PR with that change, assuming this solution is acceptable (It works fine when testing this change)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the report-processing path for setup, call, and teardown, then trace how the rerun_only exception filter decides whether a test is rerun. Reproduce the fixture-teardown MyCustomError example and verify that teardown-triggered reruns are included in the final outcome rather than reported as failures from earlier runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.