pytest-dev / pytest-dev/pytest

Teardown order mismatch with parametrized parent fixtures and scopes

Open
#12,134 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: fixtures
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Found during discussion of #11833 (I have reduced the size of the repro since I posted it in https://github.com/pytest-dev/pytest/pull/11833#issuecomment-1999818465)

import pytest

@pytest.fixture(scope="module", params=["a", "b"])
def fixture_1(request):
    print("setup 1", request.param)
    yield
    print("\nteardown 1", request.param)

@pytest.fixture(scope="module")
def fixture_2():
    print("setup 2")
    yield
    print("teardown 2")

def test_1(fixture_1, fixture_2): ...

output:

setup 1  a
setup 2
.
teardown 1 a
setup 1  b
.
teardown 1 b
teardown 2 # 2 is torn down last, perhaps surprisingly

If we reverse the order of the fixture parameters to test_1 we get an ordering where the stack is fully respected:

def test_1(fixture_2, fixture_1): ...
setup 2
setup 1 a
.
teardown 1 a
setup 1 b
.
teardown 1 b
teardown 2

What's happening in the first example is relatively straightforward -

  1. test_1 requests fixture_2, which is set up and its finalizer is queued.
  2. test_1 requests fixture_1[a], which is set up and its finalizer is queued.
  3. Test runs
  4. Because fixture_1 is parametrized, we run the test again.
  5. test_1 requests fixture_2 - and since it has module scope the value is cached, so no setup is run (and since #11833 no finalizer is queued, not that it would make any difference here)
  6. test_1 requests fixture_1[b], it has a different cache key, so fixture_1[a] is torn down, and fixture_1[b] is setup with its finalizer queued.
  7. Test finishes, and module is done, so we run the finalizers in reverse order they were added - which means fixture_1[b] first, then fixture_2, (and finally fixture_1[a] is attempted as its still on the stack, but thats also irrelevant here).

I'm not even sure this is a bug per se, but I could at least see it being surprising to users as the documentation does not imply that the stack order is ever disrespected:

https://docs.pytest.org/en/8.0.x/how-to/fixtures.html#note-on-finalizer-order

Finalizers are executed in a first-in-last-out order. For yield fixtures, the first teardown code to run is from the right-most fixture, i.e. the last test parameter.

https://docs.pytest.org/en/8.0.x/how-to/fixtures.html#yield-fixtures-recommended

Once the test is finished, pytest will go back down the list of fixtures, but in the reverse order, taking each one that yielded, and running the code inside it that was after the yield statement.

Possible fixes:

  1. When interpreting the fixture parameters, magically reorder fixtures such that stack teardown ordering is not broken - in our example turning the first example into the second.
    • This would probably be quite disruptive, and sounds very hard to code, so I don't think it's an option.
  2. Emit a warning when teardown ordering is possibly violated. Not super trivial to code, but when it's only controlling whether a warning is shown or not it needn't be perfect.
  3. Mention in the documentation that teardown order is not guaranteed to be the reverse, in the case of larger-scoped fixtures + parametrization.
  4. Also tear down fixture_2 when fixture_1[a] is torn down.

I don't think this is a huge problem though, as I think in most practical cases an end user can work around this issue by changing fixture order, make fixture_2 depend on fixture_1, change scoping, etc, which makes me favor option 3 and possibly also 2.

Related to #4871

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reduced Python fixture reproduction in the issue and reading the linked fixture-order documentation. Review discussion in #11833 and related issue #4871 before determining whether the resolution is a warning, documentation clarification, or teardown behavior change. Done requires an agreed resolution and corresponding tests or documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.