microsoft / microsoft/playwright-pytest
Test collection order broken when using page fixture
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 90
- Avg merge
- 11m
- Merged PRs (30d)
- 1
Description
Description
Given a folder contains multiple test modules each of which defining a test function that references the page fixture and one parametrized variable, when calling pytest --collect-only, then the list of test cases is not in module order but shuffled at seemingly random.
The issue occurs since pytest version 8.4.0. Tested with python 3.13.7, playwright 1.56.0 and pytest-playwright 0.7.1.
How to reproduce
pip install playwright pytest pytest-playwright
test_a.py
import pytest
@pytest.mark.parametrize("param", ["apple", "orange", "pear"])
def test_1(page, param):
assert True
test_b.py
import pytest
@pytest.mark.parametrize("param", ["dog", "cat", "mouse"])
def test_2(page, param):
assert True
Calling pytest --collect-only results in
<Package tests>
<Module test_a.py>
<Function test_1[chromium-apple]>
<Module test_b.py>
<Function test_2[chromium-dog]>
<Module test_a.py>
<Function test_1[chromium-orange]>
<Module test_b.py>
<Function test_2[chromium-cat]>
<Module test_a.py>
<Function test_1[chromium-pear]>
<Module test_b.py>
<Function test_2[chromium-mouse]>
Note: After removing the page fixture from one of the tests, the collection order is as expected:
<Package tests>
<Module test_a.py>
<Function test_1[apple]>
<Function test_1[orange]>
<Function test_1[pear]>
<Module test_b.py>
<Function test_2[chromium-dog]>
<Function test_2[chromium-cat]>
<Function test_2[chromium-mouse]>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with test_a.py and test_b.py by running pytest --collect-only, comparing collection with and without the page fixture. Trace the plugin's collection behavior and verify that the completed fix keeps all parametrized cases grouped in module order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- playwright, python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100