pytest-dev / pytest-dev/pytest

Inconsistent wasxfail value when combining pytest.mark.xfail fixture with explicit calls to pytest.xfail or pytest.fail

Open
#14,090 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

We get inconsistent wasxfail values when combining pytest.mark.xfail fixture with explicit calls to pytest.xfail or pytest.fail.
The mark.xfail reason gets ignored in the presence of those explicit calls.

Use the following conftest.py file

import pytest

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if call.when == "call":
        wasxfail = getattr(report, "wasxfail", None)
        msg = None
        if hasattr(call.excinfo, "value") and hasattr(call.excinfo.value, "msg"):
            msg = call.excinfo.value.msg
        print("\ntest: ", item.name)
        print("outcome: ", report.outcome)
        print("msg: ", msg)
        print("wasxfail: ", wasxfail)

Use the following test file

import pytest

@pytest.mark.xfail(reason="reason mark.xfail")
def test_func1():
    pytest.fail("reason pytest.fail")

@pytest.mark.xfail(reason="reason mark.xfail")
def test_func2():
    pytest.fail()

@pytest.mark.xfail(reason="reason mark.xfail")
def test_func3():
    pytest.xfail("reason pytest.xfail")

@pytest.mark.xfail(reason="reason mark.xfail")
def test_func4():
    pytest.xfail()

Result:

test:  test_func1
outcome:  skipped
msg:  reason pytest.fail
wasxfail:  reason pytest.fail         => should be "reason mark.xfail" ??

test:  test_func2
outcome:  skipped
msg:
wasxfail:                             => empty string. why? should be "reason mark.xfail"

test:  test_func3
outcome:  skipped
msg:  reason pytest.xfail
wasxfail:  reason pytest.xfail        => should be "reason mark.xfail" ??

test:  test_func4
outcome:  skipped
msg: 
wasxfail:                             => empty string. why? should be "reason mark.xfail"

See PR #14091

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the four cases from the provided conftest.py hook and test file, comparing the reported wasxfail value with the mark.xfail reason. Then inspect pytest's xfail reporting path and review PR #14091; done means the documented combinations produce consistent, expected wasxfail values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.