pytest-dev / pytest-dev/pytest-xdist
Test collection order inconsistent when parameter options come from sets
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
Test collection fails for me when using pytest-xdist:
$ pytest -n10
====================================================== test session starts ======================================================
platform linux -- Python 3.8.4, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mth/bb/softfab, configfile: pyproject.toml
plugins: xdist-1.32.0, cov-2.10.1
gw0 [201] / gw1 [201] / gw2 [201] / gw3 [201] / gw4 [201] / gw5 [201] / gw6 [201] / gw7 [201] / gw8 [201] / gw9 [201]
collecting 0 items / 1 error
============================================================ ERRORS =============================================================
_____________________________________________________ ERROR collecting gw0 ______________________________________________________
Different tests were collected between gw2 and gw0. The difference is:
--- gw2
+++ gw0
@@ -1,5 +1,5 @@
+tests/unit/test_commandline.py::test_create_user[user]
tests/unit/test_commandline.py::test_create_user[guest]
-tests/unit/test_commandline.py::test_create_user[user]
tests/unit/test_commandline.py::test_create_user[operator]
tests/unit/test_commandline.py::test_create_user[None]
tests/unit/test_commandline.py::test_remove_user
Similar diffs occur on the other workers as well.
There are more parameterized tests, but only test_create_user triggers this problem. The reason is that the four test cases after parameter substitution are not always in the same order because the parameter options originate from a set:
roleNames = {'guest', 'user', 'operator'}
@mark.parametrize('role', list(roleNames) + [None])
def test_create_user(cli, role):
...
If I change list(roleNames) to sorted(roleNames) or export PYTHONHASHSEED on the command line, the order of the options will always be the same and the problem goes away.
A possible solution would be to set PYTHONHASHSEED to the same value for all workers.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing with pytest -n10 and inspect collection behavior around tests/unit/test_commandline.py::test_create_user, whose role parameters come from a set. Done means workers collect the same test order without requiring users to sort parameters or set PYTHONHASHSEED.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100