pytest-dev / pytest-dev/pytest-xdist

Preventing oversubscription with multithreaded code?

Open
#1,012 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement needs information
Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

This is probably a feature request.

I've been finding that running pytest -n auto can take or even just seem to hang when testing one of my codebases. I believe the culprit is that a number of tests using matrix multiplication or parallelized numba functions are being hit at the same time. Both of these cases default to using the number of cores as the default number of threads – so my CPU becomes heavily oversubscribed.

Would it be possible for pytest-xdist to use something like threadpoolctl to limit the number of threads each worker uses? Ideally it could be similar to how joblib or dask set the number of threads available to each worker to something like hardware_threads // num_workers.

Alternatively, is there a good way I could set this behaviour myself? Ideally without hardcoding the number of threads to use.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named. Start by reproducing the behavior with pytest -n auto and inspect how pytest-xdist configures worker processes and parallel test execution. Done means providing a documented or implemented way to prevent thread oversubscription without hardcoding the worker count.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.