pytest-dev / pytest-dev/pytest
Update docs on parametrization gotchas
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
As https://github.com/pytest-dev/pytest/issues/1075 and other issues show it is common, when writing parametrized tests, for users to put dynamic variables (such as current time) or build the values list from structures without deterministic ordering. This causes problems with xdist because all workers (which live in different processes) need to collect the exact test items, both in order and node id.
The parametrize docs should have a section titled "Parametrization and xdist" (or something like that) that explains this problem, with examples of possible causes:
- Using non-deterministic parametrization values (for example
datetime.utcnow()): each worker might get a slightly different value, which will result in different node ids. - Using non-deterministic order: obtaining the values from a data structure which doesn't guarantee stable ordering (for example
dicton <=py35).
The section should also contain an example traceback, specially the "Different tests were collected between..." phrase, so users can easily find the section by searching.
cc @rawrgulmuffins
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the parametrization docs linked in the issue and locate the section covering parametrized tests. Add a section on parametrization and xdist with the two nondeterminism examples and a traceback containing “Different tests were collected between...”; the documentation should make the collection requirement and causes easy to find.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, testing
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100