pytest-dev / pytest-dev/pytest
Inconsistent wasxfail value when combining pytest.mark.xfail fixture with explicit calls to pytest.xfail or pytest.fail
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
We get inconsistent wasxfail values when combining pytest.mark.xfail fixture with explicit calls to pytest.xfail or pytest.fail.
The mark.xfail reason gets ignored in the presence of those explicit calls.
Use the following conftest.py file
import pytest
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
report = outcome.get_result()
if call.when == "call":
wasxfail = getattr(report, "wasxfail", None)
msg = None
if hasattr(call.excinfo, "value") and hasattr(call.excinfo.value, "msg"):
msg = call.excinfo.value.msg
print("\ntest: ", item.name)
print("outcome: ", report.outcome)
print("msg: ", msg)
print("wasxfail: ", wasxfail)
Use the following test file
import pytest
@pytest.mark.xfail(reason="reason mark.xfail")
def test_func1():
pytest.fail("reason pytest.fail")
@pytest.mark.xfail(reason="reason mark.xfail")
def test_func2():
pytest.fail()
@pytest.mark.xfail(reason="reason mark.xfail")
def test_func3():
pytest.xfail("reason pytest.xfail")
@pytest.mark.xfail(reason="reason mark.xfail")
def test_func4():
pytest.xfail()
Result:
test: test_func1
outcome: skipped
msg: reason pytest.fail
wasxfail: reason pytest.fail => should be "reason mark.xfail" ??
test: test_func2
outcome: skipped
msg:
wasxfail: => empty string. why? should be "reason mark.xfail"
test: test_func3
outcome: skipped
msg: reason pytest.xfail
wasxfail: reason pytest.xfail => should be "reason mark.xfail" ??
test: test_func4
outcome: skipped
msg:
wasxfail: => empty string. why? should be "reason mark.xfail"
See PR #14091
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the four cases from the provided conftest.py hook and test file, comparing the reported wasxfail value with the mark.xfail reason. Then inspect pytest's xfail reporting path and review PR #14091; done means the documented combinations produce consistent, expected wasxfail values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100