pytest-dev / pytest-dev/pytest-xdist

GNU Make Jobserver support?

Open
#1,283 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

I was reading https://blogs.gentoo.org/mgorny/2025/11/30/one-jobserver-to-rule-them-all/ on global / system jobservers (using the gnu make jobserver protocol as it has fairly wide support) and it noted upon xdist missing that.

It's definitely an issue I have observed when i mix workloads (e.g. multiple large test suites run in parallel, plus native compilation on the side), currently I try to eyeball my CPU graph to see how I'm doing and if I have more headroom but it would definitely be useful to be able to use jobservers to allow workers to coordinate.

https://github.com/tommilligan/pytest-jobserver exists, however it notes that it is not able to call into existing jobservers because the xdist protocol does not support sending jobserver info, so I'm assuming this requires updates to xdist itself. It also does not support the "windows jobserver" method.

Further jobserver resources:

Of note: as the essay calls out, while the jobserver protocol calls out for a fifo this is probably something to either not check or extend: in order to mitigate common failure modes, the system-wide jobserver implementations use other file types (regular file via FUSE or character device via CUSE). Make itself does not check the file type.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the xdist protocol and the limitations described by pytest-jobserver, then compare the GNU Make and Windows jobserver resources linked in the issue. Define the protocol, platform support, and worker coordination requirements before implementation; done should include an agreed approach for xdist to use existing jobservers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.