pytest-dev / pytest-dev/pytest

Calling pytest.skip isn't reflected in test outcome in make_report hook

Open
#7,125 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: reporting type: bug
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Thank you for your efforts developing pytest.
When I call pytest.skip() in pytest_runtest_setup then wait for the test's outcome in pytest_runtest_makereport the skipped flag isn't set to true and the test is marked as passed.

is this the expected behaviour?

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item: pytest.Item, call: _pytest.runner.CallInfo):
    outcome = yield
    outcome = outcome.get_result()

    # We're not interested in doing anything before running the test.
    if outcome.when == "setup":
        return

    print("\nTEST [report]", get_test_name(item), outcome.outcome)

def pytest_runtest_setup(item: pytest.Item):
    print("\nTEST [setup]:", get_test_name(item))
    pytest.skip()

def get_test_name(test: pytest.Item) -> str:
    return test.originalname or test.name

sample output, my expectation was that passed should be skipped

app/lab1/test_run_clients.py::TestTftpClientTx::test_client_send_wrq[4591_4772] 
TEST [setup]: test_client_send_wrq
SKIPPED
TEST [report] test_client_send_wrq passed

app/lab1/test_run_clients.py::TestTftpClientTx::test_client_send_data[4591_4772] 
TEST [setup]: test_client_send_data
SKIPPED
TEST [report] test_client_send_data passed

app/lab1/test_run_clients.py::TestTftpClientTx::test_client_send_ack[4591_4772] 
TEST [setup]: test_client_send_ack
SKIPPED
TEST [report] test_client_send_ack passed

pip list (relevant packages)

pylint             2.4.4  
pyparsing          2.4.7  
pytest             5.4.1  
pytest-forked      1.1.3  
pytest-timeout     1.3.4  
pytest-xdist       1.31.0 

command used to run the test (in case it's relevant)

pytest -svv --tb=short -rspx -k TestTftpClientTx app/lab1/test_run_clients.py

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The reproducer uses pytest_runtest_setup and pytest_runtest_makereport; start by running it with the reported pytest 5.4.1 command and inspect the setup report. Determine whether the passed/skipped outcome is expected, then capture the agreed behavior in a regression test or documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.