pytest-dev / pytest-dev/pytest

Unexpected behavior of skipping entire module when calling unittest.skip("a")("b")

Open
#10,821 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin: unittest status: help wanted type: bug
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

If aliasing unittest.skip with a value and inadvertently trying to pass an argument to the aliased copy, pytest will skip the entire module while unittest will throw an error.

import unittest

broken = unittest.skip("broken")


class MyTests(unittest.TestCase):
    @broken
    def test_good(self):
        assert False

    @broken("bar")
    def test_evil(self):
        assert False

pytest output

=============================================== test session starts ===============================================
platform darwin -- Python 3.11.2, pytest-7.2.2, pluggy-1.0.0
rootdir: ...
collected 0 items / 1 skipped

=============================================== 1 skipped in 0.01s ================================================

While running the same module via python -m unittest throws a unittest.case.SkipTest: broken error and fails loudly.

While the pattern of using the alias with a standard reason isn't a great pattern itself, nonetheless the behavior when it's used incorrectly is unprotected and easy to miss when you are trying to stable a test suite.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducer in the issue and compare pytest's handling of unittest.TestCase methods decorated with unittest.skip to python -m unittest. Trace pytest's collection and skip handling for the aliased decorator misuse; done means the incorrect call no longer silently skips the entire module and is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.