pytest-dev / pytest-dev/pytest

Update docs on parametrization gotchas

Open
#4,101 1 comment 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: parametrize type: docs
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

As https://github.com/pytest-dev/pytest/issues/1075 and other issues show it is common, when writing parametrized tests, for users to put dynamic variables (such as current time) or build the values list from structures without deterministic ordering. This causes problems with xdist because all workers (which live in different processes) need to collect the exact test items, both in order and node id.

The parametrize docs should have a section titled "Parametrization and xdist" (or something like that) that explains this problem, with examples of possible causes:

  • Using non-deterministic parametrization values (for example datetime.utcnow()): each worker might get a slightly different value, which will result in different node ids.
  • Using non-deterministic order: obtaining the values from a data structure which doesn't guarantee stable ordering (for example dict on <=py35).

The section should also contain an example traceback, specially the "Different tests were collected between..." phrase, so users can easily find the section by searching.

cc @rawrgulmuffins

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the parametrization docs linked in the issue and locate the section covering parametrized tests. Add a section on parametrization and xdist with the two nondeterminism examples and a traceback containing “Different tests were collected between...”; the documentation should make the collection requirement and causes easy to find.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, testing
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.