pytest-dev / pytest-dev/pytest-xdist
loadscope incorrectly modifying test ordering, when ordering is relevant
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
I'm not sure if there are other use cases, but the relevant change I'm talking about is a byproduct of #778.
With #778, we are now ordering tests based on the number of tests available. The problem is that this changes the input test order, and in some cases, such as when running with pytest-django, the test ordering is relevant. In particular, Django (and also pytest-django) orders its tests by grouping them:
(django) TestCase
TransactionalTestCase, SimpleTestCase
All other tests
Tests within each group doesn't need to be ordered, but the group ordering is relevant because TransactionalTestCase may have undesirable side-effects.
For simplicity, assume we run with one worker with --dist loadscope.
So if I have tests like this, for example:
class TestTransactionalTestCase1(TransactionalTestCase):
def test_tttc_1...
def test_tttc_2...
class TestCase1(TestCase):
def test_tc_3...
When running with 1 worker, we would get this ordering:
test_tttc_1
test_tttc_2
test_tc_3
Whereas when we run without pytest-xdist, we would run in this order (and this is the "correct" behavior based on Django's ordering):
test_tc_3
test_tttc_1
test_tttc_2
More broadly, I think re-ordering the input test ordering may conflict with other scenarios where ordering does matter. It's not a Django-specific issue, although I don't know of any other examples where this is relevant.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the --dist loadscope path and reproduce the one-worker example from the issue, comparing its order with a run without pytest-xdist. Trace where the input collection is reordered and verify behavior with pytest-django-style grouped tests. Done means loadscope no longer changes ordering that the input suite relies on while still supporting its distribution behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100