pytest-dev / pytest-dev/pytest

Fixture order considering global states

Open
#4,892 5 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: fixtures
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Hi,

This section of the documentation automatic grouping of tests by fixtures instances almost solves one need we are having, related to fixtures dealing with global states.

Problem example

To exemplify, let's simplify the example on the documentation, with seemly independent tests, each one with its fixture (modarg with test_1; and otherarg with test_2).

modarg and otherarg change the same global state.

# content of test_module.py
import pytest

@pytest.fixture(scope="module", params=['a', 'b'])
def modarg(request):
    return request.param

@pytest.fixture(scope="module", params=['a', 'b'])
def otherarg(request):
    return request.param

def test_0(otherarg):
    print("  RUN test0 with otherarg %s" % otherarg)

def test_1(modarg):
    print("  RUN test1 with modarg %s" % modarg)

It is not a problem because pytest orders in such a way that all tests using one fixture are executed, then all tests using the another:

test_example2.py::test_0[a] PASSED
test_example2.py::test_0[b] PASSED
test_example2.py::test_1[a] PASSED
test_example2.py::test_1[b] PASSED

Now, let's say I want to refactor the ['a', 'b'] parameters into a common fixture.
The problem now is that pytest tries to first use the new, highest level uppermost fixture first, now interleaving the execution of the fixtures modarg and otherarg (which is a problem because they are module scoped and are dealing with the same shared resource).

# content of test_module.py
import pytest

@pytest.fixture(scope="module", params=['a', 'b'])
def uppermost(request):
    return request.param

@pytest.fixture(scope="module")
def modarg(uppermost):
    return uppermost

@pytest.fixture(scope="module")
def otherarg(uppermost):
    return uppermost

def test_0(otherarg):
    print("  RUN test0 with otddherarg %s" % otherarg)

def test_1(modarg):
    print("  RUN test1 with modarg %s" % modarg)
test_example.py::test_0[a] PASSED
test_example.py::test_1[a] PASSED
test_example.py::test_0[b] PASSED
test_example.py::test_1[b] PASSED

Needed behaviour

In our situation, the resource is a piece of hardware, and the module-scoped fixture is an expensive set-up of that hardware (start-up and configuration).

For this case, it would solve to be able to specify that a given fixture share a resource - so, when ordering, pytest can make sure that there is no interleaving between the execution of tests that share that same resource.

So in our example, we would possibly have this:

@pytest.fixture(scope="module", shared_resource="myhardware")
def modarg(uppermost):
    return uppermost

@pytest.fixture(scope="module", shared_resource="myhardware")
def otherarg(uppermost):
    return uppermost

And that's it. I would like to hear if this is sensible to add or not, probably more experienced developers might spot any detrimental side effects of this. I would be willing to implement this myself once we decide it could and should be done.

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked documentation section on automatic grouping of tests by fixture instances and reproduce the two ordering examples from the issue. Then inspect pytest's fixture ordering and parametrization behavior to determine where shared-resource semantics would belong. Done means a decided design, implementation, and tests showing that fixtures sharing a resource are not interleaved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.