pytest-dev / pytest-dev/pytest
pytest_generate_tests cannot modify fixtures of parametrized tests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
In Matplotlib, we have a general autouse fixture that may or may not load a backend (i.e., UI toolkit) that causes a fatal error (if it doesn't exist), depending on test markers. In an effort to avoid that, I tried to add a fixture that skips tests that might use an unavailable backend. Because file-specific fixtures run after those in conftest.py, I used pytest_generate_tests to implement it, but this no longer seems to work on parametrized tests.
For the sake of an example, start with an autouse fixture that always fails in conftest.py:
# conftest.py
import pytest
@pytest.fixture(autouse=True)
def settings(request):
print('settings', request.node.fixturenames)
raise RuntimeError("This should not be called.")
then in test_foo.py, a fixture that should skip the test, inserted first via pytest_generate_tests so that it occurs before the above exception:
# test_foo.py
import pytest
def pytest_generate_tests(metafunc):
print('pytest_generate_tests')
metafunc.fixturenames.insert(0, 'should_skip')
@pytest.fixture
def should_skip():
print('should_skip')
pytest.skip('Should be skipped')
def test_foo():
print('test_foo')
assert True
This works for test_foo; should_skip is called first and the remaining fixtures are skipped.
If I then add a parametrized test:
@pytest.mark.parametrize('bar', [1, 2, 3])
def test_bar(bar):
print('test_bar', bar)
assert True
all of these call settings first instead of should_skip. The printout shows that should_skip is no longer in the list of fixtures either.
This used to work in pytest 3.6.3, but fails in 3.7.0. I bisected the issue back to #3629, but I'm not sure exactly if that intended for this behaviour to change.
$ pip list
Package Version Location
execnet 1.5.0
py 1.6.0
pytest 3.6.3.dev34+g1dc5e97a
pytest-cov 2.6.0
pytest-faulthandler 1.5.0
pytest-flake8 1.0.2
pytest-forked 0.2
pytest-rerunfailures 4.1
pytest-xdist 1.23.0
</details>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the behavior using the shown conftest.py and test_foo.py examples, comparing an unparameterized test with test_bar and the pytest 3.6.3 versus 3.7.0 behavior. Start by tracing pytest_generate_tests and fixture-name handling around the change in #3629. Done means establishing whether the regression is intentional and restoring the expected skip ordering for parametrized tests, with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100