dropbox / dropbox/pytest-flakefinder

flakefinder fails to collect tests generated by `@pytest.mark.parametrize`

Open
#18 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
158
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Long time user of flakefinder here - thank you for providing this awesome workaround for the silly pytest limitation. I have run into this extension failing with a specific type of tests. Please allow me to walk you through the issue:

Let's write a test that uses `@pytest.mark.parametrize`
```
$ cat << EOT >> test_1.py
import pytest
@pytest.mark.parametrize(
"a, b",
[(1, 2), (3, 4)],
)
def test_flake(a, b): pass
EOT
```

```
$ pytest --disable-warnings --collect-only -q test_1.py
test_1.py::test_flake[3-4]
test_1.py::test_flake[1-2]

2 tests collected in 1.31s
```
Let's run one of them:
```
$ pytest test_1.py::test_flake[1-2]
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=1277854280
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 1 item

test_1.py . [100%]

=========================================================== 1 passed in 1.31s ============================================================
```
Let's use flakefinder:

```
$ pytest --flake-finder --flake-runs=5 test_1.py::test_flake[1-2]
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=3718677389
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 0 items

========================================================= no tests ran in 1.29s ==========================================================
ERROR: not found: /mnt/nvme0/code/huggingface/m4-bs-rampup/test_1.py::test_flake[1-2]
(no name '/mnt/nvme0/code/huggingface/m4-bs-rampup/test_1.py::test_flake[1-2]' in any of [])
```
this breaks, because I think flakefinder internally uses `parametrize`:

https://github.com/dropbox/pytest-flakefinder/blob/f6fb30fac91d6e85b6a712cf2a1266654a6c14e7/pytest_flakefinder.py#L68-L71

which clashes with the test's `parametrize`

Note that tests that use a unitest `parameterized` have no problem as there it uses a different format like `test_flake_1_2`

I tried to fix this and it appears that if I immediately `return` from the overridden `pytest_generate_tests` this starts working just fine, including duplicating

```
$ pytest --flake-finder --flake-runs=5 test_1.py::test_flake[1-2]
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=1049050234
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 1 item

test_1.py .....

================================================================= PASSES =================================================================
======================================================== short test summary info =========================================================
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
=========================================================== 5 passed in 1.28s ============================================================
```

But I'm not sure how inside `pytest_generate_tests` to tell if a test has a `@pytest.mark.parametrize` decorator, I think the fix should be something like:

```
@pytest.hookimpl(tryfirst=True)
def pytest_generate_tests(self, metafunc):
"""For all true pytest tests use metafunc to add all the duplicates."""
if metafunc.has_parametrize_decorator???: # no idea how to test for this
return

fixture_name = "__flakefinder_{}".format(metafunc.function.__name__)
metafunc.fixturenames.append(fixture_name)
metafunc.parametrize(
argnames=fixture_name,
argvalues=list(range(self.flake_runs)),
)
fixture_name = "__flakefinder_{}".format(metafunc.function.__name__)
metafunc.function._pytest_duplicated = True
```

except for the weird collection report with minus values:
```
collected 2 items / 5 deselected / -3 selected
```

Also if I don't use the specific subtest it works too with the original code:

```
$ pytest --flake-finder --flake-runs=5 test_1.py
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=3507953969
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 2 items

test_1.py ..........

================================================================= PASSES =================================================================
======================================================== short test summary info =========================================================
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
=========================================================== 10 passed in 1.29s ===========================================================
```

But I need to repeat a specific sub-test.

And I can't use `pytest --flake-finder --flake-runs=5 test_1.py -k 1-2` either, because I have other tests that have the same params so it'd run other tests.

The clash happens between the specific:

```
test_1.py::test_flake[1-2]
```
and 2nd parametrize:
```
test_1.py::test_flake[1-1-2]
```
and that's why they don't match.

Thank you.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.