pytest-dev / pytest-dev/pytest-xdist

pytest-randomly seed is not respected when using loadgroup dist

Open
#1,055 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

Based on this issue: https://github.com/pytest-dev/pytest-xdist/issues/229 we expected randomly to play well with xdist

Steps:
Repeatedly invoke pytest with the following args:
--dist=loadgroup --randomly-seed=1234567890"

Expected:
Each worker runs the same set of tests in the same order.

Actual:
Each worker starts off by running the same tests, but diverges increasingly over the course of the run.

We'd really love a way to combine parallelization and determinism (while also respecting our xdist_group requirements). Determinism is critical for tracking down tests that flake due to cross test dependency. As of right now we're going to turn off parallelization all together and pay the run time cost, just to make sure that we can always reproduce failure cases.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with pytest using --dist=loadgroup --randomly-seed=1234567890 and compare the tests each worker runs and their order. Read the linked pytest-dev/pytest-xdist issue for the expected interaction between randomly and xdist_group; done means workers remain deterministic while preserving loadgroup requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.