Unit Tests are order-dependent
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 598
- Forks
- 168
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 86
Description
If executed in different order, there is a different set of failures at each run.
To reproduce, use the pytest-random-order plugin and run with randomization enabled.
While not critical, this means that tests are interfering with each other. This can be a headache when dealing with unit tests, because things may fail (or succeed) for hard to track reasons.
My proposal is the we gradually fix this by using the pytest-random-order with a seed.
We eventually change the seed and fix the leakings it reveals. A possiblity is to schedule CI to open PR's updating the seed.
When re-running the suite with randomized order does not cause failures anymore, we stop using a seed and leave random order by default.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the Python test suite with pytest-random-order and a fixed seed, as described in the issue, then trace each order-dependent failure to the interfering tests. Done means randomized runs no longer cause failures, after which the seed-based process can be removed; no source or test file is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100