pytest-dev / pytest-dev/pytest

pytest_generate_tests cannot modify fixtures of parametrized tests

Open
#4,017 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: fixtures topic: parametrize type: refactoring
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

In Matplotlib, we have a general autouse fixture that may or may not load a backend (i.e., UI toolkit) that causes a fatal error (if it doesn't exist), depending on test markers. In an effort to avoid that, I tried to add a fixture that skips tests that might use an unavailable backend. Because file-specific fixtures run after those in conftest.py, I used pytest_generate_tests to implement it, but this no longer seems to work on parametrized tests.

For the sake of an example, start with an autouse fixture that always fails in conftest.py:

# conftest.py
import pytest

@pytest.fixture(autouse=True)
def settings(request):
    print('settings', request.node.fixturenames)
    raise RuntimeError("This should not be called.")

then in test_foo.py, a fixture that should skip the test, inserted first via pytest_generate_tests so that it occurs before the above exception:

# test_foo.py
import pytest

def pytest_generate_tests(metafunc):
    print('pytest_generate_tests')
    metafunc.fixturenames.insert(0, 'should_skip')

@pytest.fixture
def should_skip():
    print('should_skip')
    pytest.skip('Should be skipped')

def test_foo():
    print('test_foo')
    assert True

This works for test_foo; should_skip is called first and the remaining fixtures are skipped.

If I then add a parametrized test:

@pytest.mark.parametrize('bar', [1, 2, 3])
def test_bar(bar):
    print('test_bar', bar)
    assert True

all of these call settings first instead of should_skip. The printout shows that should_skip is no longer in the list of fixtures either.

This used to work in pytest 3.6.3, but fails in 3.7.0. I bisected the issue back to #3629, but I'm not sure exactly if that intended for this behaviour to change.

Hopefully all the relevant info: ``` $ python --version Python 3.6.6

$ pip list
Package Version Location


execnet 1.5.0
py 1.6.0
pytest 3.6.3.dev34+g1dc5e97a
pytest-cov 2.6.0
pytest-faulthandler 1.5.0
pytest-flake8 1.0.2
pytest-forked 0.2
pytest-rerunfailures 4.1
pytest-xdist 1.23.0

</details>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the behavior using the shown conftest.py and test_foo.py examples, comparing an unparameterized test with test_bar and the pytest 3.6.3 versus 3.7.0 behavior. Start by tracing pytest_generate_tests and fixture-name handling around the change in #3629. Done means establishing whether the regression is intentional and restoring the expected skip ordering for parametrized tests, with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.