pytest-dev / pytest-dev/pytest-xdist

Test collection order inconsistent when parameter options come from sets

Open
#598 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

Test collection fails for me when using pytest-xdist:

$ pytest -n10 
====================================================== test session starts ======================================================
platform linux -- Python 3.8.4, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mth/bb/softfab, configfile: pyproject.toml
plugins: xdist-1.32.0, cov-2.10.1
gw0 [201] / gw1 [201] / gw2 [201] / gw3 [201] / gw4 [201] / gw5 [201] / gw6 [201] / gw7 [201] / gw8 [201] / gw9 [201]
collecting 0 items / 1 error                                                                                                    
============================================================ ERRORS =============================================================
_____________________________________________________ ERROR collecting gw0 ______________________________________________________
Different tests were collected between gw2 and gw0. The difference is:
--- gw2

+++ gw0

@@ -1,5 +1,5 @@

+tests/unit/test_commandline.py::test_create_user[user]
 tests/unit/test_commandline.py::test_create_user[guest]
-tests/unit/test_commandline.py::test_create_user[user]
 tests/unit/test_commandline.py::test_create_user[operator]
 tests/unit/test_commandline.py::test_create_user[None]
 tests/unit/test_commandline.py::test_remove_user

Similar diffs occur on the other workers as well.

There are more parameterized tests, but only test_create_user triggers this problem. The reason is that the four test cases after parameter substitution are not always in the same order because the parameter options originate from a set:

roleNames = {'guest', 'user', 'operator'}

@mark.parametrize('role', list(roleNames) + [None])
def test_create_user(cli, role):
    ...

If I change list(roleNames) to sorted(roleNames) or export PYTHONHASHSEED on the command line, the order of the options will always be the same and the problem goes away.

A possible solution would be to set PYTHONHASHSEED to the same value for all workers.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing with pytest -n10 and inspect collection behavior around tests/unit/test_commandline.py::test_create_user, whose role parameters come from a set. Done means workers collect the same test order without requiring users to sort parameters or set PYTHONHASHSEED.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.