pytest-dev / pytest-dev/pytest-xdist

test item scheduling does not schedule N items to N workers

Open
#40 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

I may just be misunderstanding how xdist works, but I can't seem to get the number of workers I requested to all accept tasks in parallel:

$ py.test -v -n 8
================================ test session starts =================================
platform darwin -- Python 3.5.1, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- .../pytest-xdist-testcase/venv/bin/python3
cachedir: .cache
rootdir: .../pytest-xdist-testcase, inifile: 
plugins: xdist-1.13.1
[gw0] darwin Python 3.5.1 cwd: .../pytest-xdist-testcase
[gw1] darwin Python 3.5.1 cwd: .../pytest-xdist-testcase
[gw2] darwin Python 3.5.1 cwd: .../pytest-xdist-testcase
[gw3] darwin Python 3.5.1 cwd: .../pytest-xdist-testcase
[gw4] darwin Python 3.5.1 cwd: .../pytest-xdist-testcase
[gw5] darwin Python 3.5.1 cwd: .../pytest-xdist-testcase
[gw6] darwin Python 3.5.1 cwd: .../pytest-xdist-testcase
[gw7] darwin Python 3.5.1 cwd: .../pytest-xdist-testcase
[gw0] Python 3.5.1 (default, Dec  7 2015, 21:59:10)  -- [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
[gw1] Python 3.5.1 (default, Dec  7 2015, 21:59:10)  -- [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
[gw2] Python 3.5.1 (default, Dec  7 2015, 21:59:10)  -- [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
[gw3] Python 3.5.1 (default, Dec  7 2015, 21:59:10)  -- [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
[gw4] Python 3.5.1 (default, Dec  7 2015, 21:59:10)  -- [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
[gw5] Python 3.5.1 (default, Dec  7 2015, 21:59:10)  -- [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
[gw6] Python 3.5.1 (default, Dec  7 2015, 21:59:10)  -- [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
[gw7] Python 3.5.1 (default, Dec  7 2015, 21:59:10)  -- [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
gw0 [8] / gw1 [8] / gw2 [8] / gw3 [8] / gw4 [8] / gw5 [8] / gw6 [8] / gw7 [8]
scheduling tests via LoadScheduling

test_tests.py::test_1[0] 
test_tests.py::test_1[2] 
test_tests.py::test_2[0] 
test_tests.py::test_2[2] 
[gw0] PASSED test_tests.py::test_1[2] 
[gw7] PASSED test_tests.py::test_1[0] 
[gw2] PASSED test_tests.py::test_2[0] 
[gw4] PASSED test_tests.py::test_2[2] 
test_tests.py::test_2[1] 
test_tests.py::test_1[3] 
test_tests.py::test_1[1] 
test_tests.py::test_2[3] 
[gw7] PASSED test_tests.py::test_1[1] 
[gw0] PASSED test_tests.py::test_1[3] 
[gw4] PASSED test_tests.py::test_2[3] 
[gw2] PASSED test_tests.py::test_2[1] 

============================= 8 passed in 64.12 seconds ==============================

Watching the output as it comes, it looks like the second block of 4 tests doesn't start running until the first block of 4 completes.

Am I misunderstanding something? Why don't all 8 workers start running tests at the same time? Why doesn't gw1, for example, get any tests?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the report with py.test -v -n 8 and the eight parametrized tests shown in the issue. Start by tracing the LoadScheduling behavior and worker assignments; done means explaining or correcting why only four tests run initially and why gw1 receives no test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.