pytest-dev / pytest-dev/pytest-xdist
Upgrading from 2.3.0 to 3.3.1 and use --dist=worksteal reorders the test cases
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
Hi,
we have a long running test suite which consists of a single test function run on different input files (the input files are passed via @pytest.mark.parametrize). The function runs longer for larger input files, therefore we are sorting the input files according to their sizes such that the largest file comes first. This has been working well with the admittedly old version with the default distribution scheduling.
Now I have read about the worksteal option and wanted to give it a try. Unfortunately, I'm realizing now that the order seems to be randomized. Because the duration of the tests might differ by some magnitudes, it is very important for a short overall duration to execute the long running tests first. Is there a way to do that with the current version?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the ordering change with pytest-xdist 3.3.1, a parametrized test, and --dist=worksteal. Then trace the worksteal scheduling entry point and existing tests for ordering behavior. Done should define and verify the intended ordering behavior, or clearly document that preserving parametrization order is unsupported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100