pytest-dev / pytest-dev/pytest
Adding scope to parametrized fixture changes test ordering, (e.g. confuses PyCharm)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
The output of pytest test_.py (specifically the order of tests) changes when using a function-scoped parametrized fixture versus any other scoped fixture.
Click to toggle MWE
# ./test_.py
import pytest
@pytest.fixture(
scope="class",
params=["one", "two"],
)
def f(request):
return request.param
def test_one(f):
assert f
def test_two(f):
assert f
Running the MWE with a function-scoped fixture produces this output:
❯ pytest -v test_.py
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.9.6, pytest-7.3.2, pluggy-1.0.0
rootdir: /Users/cleancoder/dev/scratch/pythonProject
collected 4 items
test_.py::test_one[one] PASSED [ 25%]
test_.py::test_one[two] PASSED [ 50%]
test_.py::test_two[one] PASSED [ 75%]
test_.py::test_two[two] PASSED [100%]
======================================================================== 4 passed in 0.01s =========================================================================
Changing the scope to class also changes the test ordering:
❯ pytest -v test_.py
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.9.6, pytest-7.3.2, pluggy-1.0.0
rootdir: /Users/cleancoder/dev/scratch/pythonProject
collected 4 items
test_.py::test_one[one] PASSED [ 25%]
test_.py::test_two[one] PASSED [ 50%]
test_.py::test_one[two] PASSED [ 75%]
test_.py::test_two[two] PASSED [100%]
======================================================================== 4 passed in 0.01s =========================================================================
The correct (aka expected) output allows PyCharm to group the test results nicely:
The other output confuses PyCharm (and also me when reading the results on the CLI directly):
Other details:
- Python 3.9.6
- macOS 13.4
Output of pip list:
❯ pip list
Package Version
-------------- -------
exceptiongroup 1.1.1
iniconfig 2.0.0
packaging 23.1
pip 22.3.1
pluggy 1.0.0
pytest 7.3.2
setuptools 67.0.0
tomli 2.0.1
wheel 0.38.4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the minimal example in test_.py with the fixture scoped as function and class, then compare pytest -v test_.py collection order. Trace how parametrized fixture scope affects ordering and verify that the result consistently groups each test's parameter cases as shown in the expected output, without confusing PyCharm.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100