pytest-dev / pytest-dev/pytest

Adding scope to parametrized fixture changes test ordering, (e.g. confuses PyCharm)

Open
#11,121 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: fixtures
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

The output of pytest test_.py (specifically the order of tests) changes when using a function-scoped parametrized fixture versus any other scoped fixture.

Click to toggle MWE
# ./test_.py
import pytest


@pytest.fixture(
    scope="class",
    params=["one", "two"],
)
def f(request):
    return request.param


def test_one(f):
    assert f


def test_two(f):
    assert f

Running the MWE with a function-scoped fixture produces this output:

❯ pytest -v test_.py
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.9.6, pytest-7.3.2, pluggy-1.0.0
rootdir: /Users/cleancoder/dev/scratch/pythonProject
collected 4 items                                                                                                                                                

test_.py::test_one[one] PASSED                                                                                                                  [ 25%]
test_.py::test_one[two] PASSED                                                                                                                  [ 50%]
test_.py::test_two[one] PASSED                                                                                                                  [ 75%]
test_.py::test_two[two] PASSED                                                                                                                  [100%]

======================================================================== 4 passed in 0.01s =========================================================================

Changing the scope to class also changes the test ordering:

❯ pytest -v test_.py
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.9.6, pytest-7.3.2, pluggy-1.0.0
rootdir: /Users/cleancoder/dev/scratch/pythonProject
collected 4 items                                                                                                                                                

test_.py::test_one[one] PASSED                                                                                                                  [ 25%]
test_.py::test_two[one] PASSED                                                                                                                  [ 50%]
test_.py::test_one[two] PASSED                                                                                                                  [ 75%]
test_.py::test_two[two] PASSED                                                                                                                  [100%]

======================================================================== 4 passed in 0.01s =========================================================================

The correct (aka expected) output allows PyCharm to group the test results nicely:

Screenshot 2023-06-19 at 10 33 52

The other output confuses PyCharm (and also me when reading the results on the CLI directly):

Screenshot 2023-06-19 at 10 34 03

Other details:

  • Python 3.9.6
  • macOS 13.4

Output of pip list:

❯ pip list
Package        Version
-------------- -------
exceptiongroup 1.1.1
iniconfig      2.0.0
packaging      23.1
pip            22.3.1
pluggy         1.0.0
pytest         7.3.2
setuptools     67.0.0
tomli          2.0.1
wheel          0.38.4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the minimal example in test_.py with the fixture scoped as function and class, then compare pytest -v test_.py collection order. Trace how parametrized fixture scope affects ordering and verify that the result consistently groups each test's parameter cases as shown in the expected output, without confusing PyCharm.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.