pytest-dev / pytest-dev/pytest-xdist
GNU Make Jobserver support?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
I was reading https://blogs.gentoo.org/mgorny/2025/11/30/one-jobserver-to-rule-them-all/ on global / system jobservers (using the gnu make jobserver protocol as it has fairly wide support) and it noted upon xdist missing that.
It's definitely an issue I have observed when i mix workloads (e.g. multiple large test suites run in parallel, plus native compilation on the side), currently I try to eyeball my CPU graph to see how I'm doing and if I have more headroom but it would definitely be useful to be able to use jobservers to allow workers to coordinate.
https://github.com/tommilligan/pytest-jobserver exists, however it notes that it is not able to call into existing jobservers because the xdist protocol does not support sending jobserver info, so I'm assuming this requires updates to xdist itself. It also does not support the "windows jobserver" method.
Further jobserver resources:
-
https://make.mad-scientist.net/papers/jobserver-implementation/
-
https://www.chiark.greenend.org.uk/doc/make-doc/make.html/Integrating-make.html
provides a unified view of https://www.gnu.org/software/make/manual/html_node/Job-Slots.html + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html which are the "official" docs
Of note: as the essay calls out, while the jobserver protocol calls out for a fifo this is probably something to either not check or extend: in order to mitigate common failure modes, the system-wide jobserver implementations use other file types (regular file via FUSE or character device via CUSE). Make itself does not check the file type.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the xdist protocol and the limitations described by pytest-jobserver, then compare the GNU Make and Windows jobserver resources linked in the issue. Define the protocol, platform support, and worker coordination requirements before implementation; done should include an agreed approach for xdist to use existing jobservers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100