pytest-dev / pytest-dev/pytest-xdist
Preventing oversubscription with multithreaded code?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
This is probably a feature request.
I've been finding that running pytest -n auto can take or even just seem to hang when testing one of my codebases. I believe the culprit is that a number of tests using matrix multiplication or parallelized numba functions are being hit at the same time. Both of these cases default to using the number of cores as the default number of threads – so my CPU becomes heavily oversubscribed.
Would it be possible for pytest-xdist to use something like threadpoolctl to limit the number of threads each worker uses? Ideally it could be similar to how joblib or dask set the number of threads available to each worker to something like hardware_threads // num_workers.
Alternatively, is there a good way I could set this behaviour myself? Ideally without hardcoding the number of threads to use.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named. Start by reproducing the behavior with pytest -n auto and inspect how pytest-xdist configures worker processes and parallel test execution. Done means providing a documented or implemented way to prevent thread oversubscription without hardcoding the worker count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100