pytest-dev / pytest-dev/pytest

metafunc.parametrize(..., scope="session") fails when mixed with other parametrizations

Open
#3,542 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

topic: parametrize type: bug
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Both, python 2.7.13 and 3.5.3, pytest 3.6.1, no additional pytest plugins.

# conftest.py
import pytest

KNOWN_BRANCHES = {
    # The number means how many times given branch has been set-up
    "branch_1": 0,
    "branch_2": 0,
}
def pytest_generate_tests(metafunc):
    if 'tested_branch' in metafunc.fixturenames:
        branches = KNOWN_BRANCHES.keys()
        metafunc.parametrize("tested_branch", branches, scope="session")

@pytest.fixture(scope="session")
def repo_preparation(tested_branch):
    """ If it would have a "session" scopoe then the value should be always 1 for each branch."""
    KNOWN_BRANCHES[tested_branch] += 1
    return KNOWN_BRANCHES[tested_branch]

And the simplest reproduction:

# test_parametrization.py
import pytest

def test_nothing(repo_preparation):
    assert repo_preparation == 1  # that passes

@pytest.mark.parametrize("_", ["param_1", "param_2"])
def test_basic_parametrization(repo_preparation, _):
    # that causes parametrization confisuon
    assert repo_preparation == 1, "Repo prepared more than once."

Gives such an result:

$ pytest -v test_parametrization.py 
================================== test session starts ==================================
platform linux -- Python 3.5.3, pytest-3.6.1, py-1.5.3, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/kamichal/ecws/pyplayground, inifile:
collected 6 items                                                                       

test_parametrization.py::test_nothing[branch_1] PASSED                            [ 16%]
test_parametrization.py::test_basic_parametrization[branch_1-param_1] PASSED      [ 33%]
test_parametrization.py::test_nothing[branch_2] PASSED                            [ 50%]
test_parametrization.py::test_basic_parametrization[branch_1-param_2] PASSED      [ 66%]
test_parametrization.py::test_basic_parametrization[branch_2-param_1] FAILED      [ 83%]
test_parametrization.py::test_basic_parametrization[branch_2-param_2] FAILED      [100%]

======================================= FAILURES ========================================
_____________________ test_basic_parametrization[branch_2-param_1] ______________________

repo_preparation = 2, _ = 'param_1'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 2 == 1

test_parametrization.py:12: AssertionError
_____________________ test_basic_parametrization[branch_2-param_2] ______________________

repo_preparation = 3, _ = 'param_2'

    @pytest.mark.parametrize("_", ["param_1", "param_2"])
    def test_basic_parametrization(repo_preparation, _):
        # that causes parametrization confisuon
>       assert repo_preparation == 1, "Repo prepared more than once."
E       AssertionError: Repo prepared more than once.
E       assert 3 == 1

test_parametrization.py:12: AssertionError
========================== 2 failed, 4 passed in 0.03 seconds ===========================

Setting indirect=True in the metafunc.parametrize call fixes that problem, but there is no information that it's required for larger scopes to work properly. There is only an advice to do so, but...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure using conftest.py and test_parametrization.py with pytest 3.6.1, comparing session-scoped parametrization with and without indirect=True. Trace pytest's parametrization and fixture setup behavior, then add a regression test or documentation clarification so mixed parametrizations with session scope behave as intended.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.