pytest-dev / pytest-dev/pytest

self.skipTest does not produce the same results as unittest does

Open
#13,756 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Given this test case:

from unittest import TestCase, main

class T(TestCase):

    def test_foo(self):
        for i in range(5):
            with self.subTest(msg="custom", i=i):
                if i % 2 == 0:
                    self.skipTest('even number')


if __name__ == '__main__':
    main()

Running with python:

λ python .tmp\test-ut-skip.py
sss
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK (skipped=3)

Running with pytest:

 λ pytest .tmp\test-ut-skip.py -q
ss                                                             [100%]
2 skipped in 0.01 seconds

The problem is that TestCaseFunction.addSkip will append the exception info to a list, but pytest_runtest_makereport will only pop one item off the list. We need to find a way to hook into TestCase.subTest to issue pytest_runtest_logreport from within the with statement.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the example with both unittest and pytest, then inspect TestCaseFunction.addSkip and pytest_runtest_makereport. Trace how each self.subTest skip is recorded and reported; done means pytest reports all three skipped subtests, matching unittest's result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.