pytest-dev / pytest-dev/pytest-xdist
Shut down idle workers early to free memory when no tests are left
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
When running with many workers (e.g., -n 24), some workers finish their share of tests well before others (for example, when you have a few extremely long running tests, and a lot of short-lived tests). Those idle workers sit around, not releasing the memory they used for the previous tests, for the rest of the run.
In our case we run a large simulation test suite with --maxprocesses=24 and each worker process holds onto a few hundred MB. When half the workers are done but the other half still have slow integration tests to grind through, we're wasting several GB of RAM on processes that will never run another test.
Proposed behavior
When a worker has no more tests to run and the scheduler confirms it won't receive any more work, shut down that worker's process instead of keeping it alive until the entire session ends.
This could be default behavior (seems like there's no reason to keep an idle worker alive), or opt-in via a flag like --early-shutdown.
Why this matters
It doesn't matter on a local machine (at least not much), But in CI, that's several GB of wasted memory on a shared machine, which could've been contributed to other teams running their tests.
It also plays well with --dist worksteal: once stealing is exhausted and a worker has nothing left. (Does it even make sense with any dist other than worksteal? Maybe load with maxschedchunk?)
I'm willing to contribute this feature, but first I'd appreciate a confirmation that this is possible, and maybe some guidance on what part of the codebase should this be injected :)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the worker and scheduler lifecycle for --dist worksteal, especially where --maxprocesses workers receive or exhaust test work; the issue does not name specific files or tests. Confirm how --maxschedchunk and other distribution modes behave, then verify that workers shut down only after the scheduler confirms no more work while the remaining tests still complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100