pytest-dev / pytest-dev/pytest
Calling pytest.skip isn't reflected in test outcome in make_report hook
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
Thank you for your efforts developing pytest.
When I call pytest.skip() in pytest_runtest_setup then wait for the test's outcome in pytest_runtest_makereport the skipped flag isn't set to true and the test is marked as passed.
is this the expected behaviour?
@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item: pytest.Item, call: _pytest.runner.CallInfo):
outcome = yield
outcome = outcome.get_result()
# We're not interested in doing anything before running the test.
if outcome.when == "setup":
return
print("\nTEST [report]", get_test_name(item), outcome.outcome)
def pytest_runtest_setup(item: pytest.Item):
print("\nTEST [setup]:", get_test_name(item))
pytest.skip()
def get_test_name(test: pytest.Item) -> str:
return test.originalname or test.name
sample output, my expectation was that passed should be skipped
app/lab1/test_run_clients.py::TestTftpClientTx::test_client_send_wrq[4591_4772]
TEST [setup]: test_client_send_wrq
SKIPPED
TEST [report] test_client_send_wrq passed
app/lab1/test_run_clients.py::TestTftpClientTx::test_client_send_data[4591_4772]
TEST [setup]: test_client_send_data
SKIPPED
TEST [report] test_client_send_data passed
app/lab1/test_run_clients.py::TestTftpClientTx::test_client_send_ack[4591_4772]
TEST [setup]: test_client_send_ack
SKIPPED
TEST [report] test_client_send_ack passed
pip list (relevant packages)
pylint 2.4.4
pyparsing 2.4.7
pytest 5.4.1
pytest-forked 1.1.3
pytest-timeout 1.3.4
pytest-xdist 1.31.0
command used to run the test (in case it's relevant)
pytest -svv --tb=short -rspx -k TestTftpClientTx app/lab1/test_run_clients.py
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The reproducer uses pytest_runtest_setup and pytest_runtest_makereport; start by running it with the reported pytest 5.4.1 command and inspect the setup report. Determine whether the passed/skipped outcome is expected, then capture the agreed behavior in a regression test or documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100