dropbox / dropbox/pytest-flakefinder
flakefinder fails to collect tests generated by `@pytest.mark.parametrize`
- Dominant language
- Python
- Stars
- 158
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
Long time user of flakefinder here - thank you for providing this awesome workaround for the silly pytest limitation. I have run into this extension failing with a specific type of tests. Please allow me to walk you through the issue:
Let's write a test that uses `@pytest.mark.parametrize`
```
$ cat << EOT >> test_1.py
import pytest
@pytest.mark.parametrize(
"a, b",
[(1, 2), (3, 4)],
)
def test_flake(a, b): pass
EOT
```
```
$ pytest --disable-warnings --collect-only -q test_1.py
test_1.py::test_flake[3-4]
test_1.py::test_flake[1-2]
2 tests collected in 1.31s
```
Let's run one of them:
```
$ pytest test_1.py::test_flake[1-2]
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=1277854280
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 1 item
test_1.py . [100%]
=========================================================== 1 passed in 1.31s ============================================================
```
Let's use flakefinder:
```
$ pytest --flake-finder --flake-runs=5 test_1.py::test_flake[1-2]
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=3718677389
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 0 items
========================================================= no tests ran in 1.29s ==========================================================
ERROR: not found: /mnt/nvme0/code/huggingface/m4-bs-rampup/test_1.py::test_flake[1-2]
(no name '/mnt/nvme0/code/huggingface/m4-bs-rampup/test_1.py::test_flake[1-2]' in any of [])
```
this breaks, because I think flakefinder internally uses `parametrize`:
https://github.com/dropbox/pytest-flakefinder/blob/f6fb30fac91d6e85b6a712cf2a1266654a6c14e7/pytest_flakefinder.py#L68-L71
which clashes with the test's `parametrize`
Note that tests that use a unitest `parameterized` have no problem as there it uses a different format like `test_flake_1_2`
I tried to fix this and it appears that if I immediately `return` from the overridden `pytest_generate_tests` this starts working just fine, including duplicating
```
$ pytest --flake-finder --flake-runs=5 test_1.py::test_flake[1-2]
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=1049050234
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 1 item
test_1.py .....
================================================================= PASSES =================================================================
======================================================== short test summary info =========================================================
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
=========================================================== 5 passed in 1.28s ============================================================
```
But I'm not sure how inside `pytest_generate_tests` to tell if a test has a `@pytest.mark.parametrize` decorator, I think the fix should be something like:
```
@pytest.hookimpl(tryfirst=True)
def pytest_generate_tests(self, metafunc):
"""For all true pytest tests use metafunc to add all the duplicates."""
if metafunc.has_parametrize_decorator???: # no idea how to test for this
return
fixture_name = "__flakefinder_{}".format(metafunc.function.__name__)
metafunc.fixturenames.append(fixture_name)
metafunc.parametrize(
argnames=fixture_name,
argvalues=list(range(self.flake_runs)),
)
fixture_name = "__flakefinder_{}".format(metafunc.function.__name__)
metafunc.function._pytest_duplicated = True
```
except for the weird collection report with minus values:
```
collected 2 items / 5 deselected / -3 selected
```
Also if I don't use the specific subtest it works too with the original code:
```
$ pytest --flake-finder --flake-runs=5 test_1.py
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
Using --randomly-seed=3507953969
rootdir: /mnt/nvme0/code/huggingface/m4-bs-rampup
plugins: randomly-3.12.0, forked-1.4.0, hydra-core-1.3.0, anyio-3.6.2, subtests-0.9.0, hypothesis-6.47.4, timeout-2.1.0, dash-2.7.1, xdist-3.1.0, instafail-0.4.2, flakefinder-1.0.0
collected 2 items
test_1.py ..........
================================================================= PASSES =================================================================
======================================================== short test summary info =========================================================
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[1-2]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
PASSED test_1.py::test_flake[3-4]
=========================================================== 10 passed in 1.29s ===========================================================
```
But I need to repeat a specific sub-test.
And I can't use `pytest --flake-finder --flake-runs=5 test_1.py -k 1-2` either, because I have other tests that have the same params so it'd run other tests.
The clash happens between the specific:
```
test_1.py::test_flake[1-2]
```
and 2nd parametrize:
```
test_1.py::test_flake[1-1-2]
```
and that's why they don't match.
Thank you.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.