pytest-dev / pytest-dev/pytest-xdist

loadscope incorrectly modifying test ordering, when ordering is relevant

Open
#1,083 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

I'm not sure if there are other use cases, but the relevant change I'm talking about is a byproduct of #778.

With #778, we are now ordering tests based on the number of tests available. The problem is that this changes the input test order, and in some cases, such as when running with pytest-django, the test ordering is relevant. In particular, Django (and also pytest-django) orders its tests by grouping them:

(django) TestCase
TransactionalTestCase, SimpleTestCase
All other tests

Tests within each group doesn't need to be ordered, but the group ordering is relevant because TransactionalTestCase may have undesirable side-effects.

For simplicity, assume we run with one worker with --dist loadscope.

So if I have tests like this, for example:

class TestTransactionalTestCase1(TransactionalTestCase):
    def test_tttc_1...

    def test_tttc_2...

class TestCase1(TestCase):
    def test_tc_3...

When running with 1 worker, we would get this ordering:

test_tttc_1
test_tttc_2
test_tc_3

Whereas when we run without pytest-xdist, we would run in this order (and this is the "correct" behavior based on Django's ordering):

test_tc_3
test_tttc_1
test_tttc_2

More broadly, I think re-ordering the input test ordering may conflict with other scenarios where ordering does matter. It's not a Django-specific issue, although I don't know of any other examples where this is relevant.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the --dist loadscope path and reproduce the one-worker example from the issue, comparing its order with a run without pytest-xdist. Trace where the input collection is reordered and verify behavior with pytest-django-style grouped tests. Done means loadscope no longer changes ordering that the input suite relies on while still supporting its distribution behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.