microsoft / microsoft/playwright-pytest

Test collection order broken when using page fixture

Open
#304 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
563
Forks
90
Avg merge
11m
Merged PRs (30d)
1

Description

Description

Given a folder contains multiple test modules each of which defining a test function that references the page fixture and one parametrized variable, when calling pytest --collect-only, then the list of test cases is not in module order but shuffled at seemingly random.

The issue occurs since pytest version 8.4.0. Tested with python 3.13.7, playwright 1.56.0 and pytest-playwright 0.7.1.

How to reproduce

pip install playwright pytest pytest-playwright

test_a.py

import pytest

@pytest.mark.parametrize("param", ["apple", "orange", "pear"])
def test_1(page, param):
    assert True

test_b.py

import pytest

@pytest.mark.parametrize("param", ["dog", "cat", "mouse"])
def test_2(page, param):
    assert True

Calling pytest --collect-only results in

<Package tests>
  <Module test_a.py>
    <Function test_1[chromium-apple]>
  <Module test_b.py>
    <Function test_2[chromium-dog]>
  <Module test_a.py>
    <Function test_1[chromium-orange]>
  <Module test_b.py>
    <Function test_2[chromium-cat]>
  <Module test_a.py>
    <Function test_1[chromium-pear]>
  <Module test_b.py>
    <Function test_2[chromium-mouse]>

Note: After removing the page fixture from one of the tests, the collection order is as expected:

<Package tests>
  <Module test_a.py>
    <Function test_1[apple]>
    <Function test_1[orange]>
    <Function test_1[pear]>
  <Module test_b.py>
    <Function test_2[chromium-dog]>
    <Function test_2[chromium-cat]>
    <Function test_2[chromium-mouse]>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with test_a.py and test_b.py by running pytest --collect-only, comparing collection with and without the page fixture. Trace the plugin's collection behavior and verify that the completed fix keeps all parametrized cases grouped in module order.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, python
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.