pytest-dev / pytest-dev/pytest

Tests not collected if parametrized with `itertools` objects and imported in multiple files

Open
#12,685 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: parametrize type: deprecation
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

I'm observing a very odd behavior of tests that are imported from other files getting collected only for one test file when they have parametrized arguments that are generated through itertools.

Example with 3 files:

  • common.py:
import pytest
import itertools

class CommonTestSuite:
    @pytest.mark.parametrize(
        "m,n",
        itertools.product(
            [100, 1_000],
            [100, 1_000],
        )
    )
    def test_no_negative_numbers(self, generator, m, n):
        assert generator(m,n).min() >= 0.
  • test_poisson.py:
from common import CommonTestSuite
import pytest
import numpy as np

class PoissonGenerator:
    def __init__(self):
        self.rng = np.random.default_rng(seed=123)
    def generate(self, m, n):
        return self.rng.poisson(1, size=(m,n))

class TestPoisson(CommonTestSuite):
    obj = PoissonGenerator()

    @pytest.fixture
    def generator(self):
        return self.obj.generate
  • test_gamma.py:
from common import CommonTestSuite
import pytest
import numpy as np

class GammaGenerator:
    def __init__(self):
        self.rng = np.random.default_rng(seed=123)
    def generate(self, m, n):
        return self.rng.standard_gamma(1, size=(m,n))

class TestGamma(CommonTestSuite):
    obj = GammaGenerator()

    @pytest.fixture
    def generator(self):
        return self.obj.generate

If I run pytest . --co, this is what I get:

============================================ test session starts ============================================
platform linux -- Python 3.11.0, pytest-7.4.0, pluggy-1.0.0
rootdir: /home/david/del/pytest_issue
plugins: anyio-3.5.0, mypy-0.10.3
collected 5 items                                                                                           

<Module test_gamma.py>
  <Class TestGamma>
    <Function test_no_negative_numbers[100-100]>
    <Function test_no_negative_numbers[100-1000]>
    <Function test_no_negative_numbers[1000-100]>
    <Function test_no_negative_numbers[1000-1000]>
<Module test_poisson.py>
  <Class TestPoisson>
    <Function test_no_negative_numbers[m0-n0]>

That is: the tests from the common class were only collected for the first file that is executed when running pytest .. Same behavior happens if I add more files importing from common.py.

If I modify the file common.py as follows (no itertools-generated parametrization):

import pytest

class CommonTestSuite:
    @pytest.mark.parametrize(
        "m,n",
        [
            (100, 10),
            (10, 100)
        ]
    )
    def test_no_negative_numbers(self, generator, m, n):
        assert generator(m,n).min() >= 0.

Then the tests get collected as expected:

platform linux -- Python 3.11.0, pytest-7.4.0, pluggy-1.0.0
rootdir: /home/david/del/pytest_issue
plugins: anyio-3.5.0, mypy-0.10.3
collected 4 items                                                                                           

<Module test_gamma.py>
  <Class TestGamma>
    <Function test_no_negative_numbers[100-10]>
    <Function test_no_negative_numbers[10-100]>
<Module test_poisson.py>
  <Class TestPoisson>
    <Function test_no_negative_numbers[100-10]>
    <Function test_no_negative_numbers[10-100]>

If instead of importing the common tests from the same file in both test_gamma.py and test_poisson.py I copy-paste the code with itertools or "import" the file by using file.read() + exec(), then the tests are collected correctly as expected.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the report with common.py, test_gamma.py, and test_poisson.py, then run pytest . --co to compare collection with the list-based parametrization. Trace how pytest handles the shared itertools parametrization and confirm that both derived test classes collect all four cases without changing the reported test behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.