feature: Support make jobserver protocol
- Dominant language
- Python
- Stars
- 140
- Forks
- 45
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
Make has already for some time used jobserver protocol such that when a sub-make is run, top-make decides whether more processes can be spawned. This is a requirement for -jN option to work with nested makes. Recently, there's a PR to add jobserver *client* support to ninja as well. I propose buildstream to implement the actual jobserver part (at least fifo, can use also pipe) and mounts relevant fifo into build sandbox. There is also the part of setting MAKEFLAGS and declaring where the job server is. When this setup is fully in place and user passes --max-jobs to BuildStream, it can actually take effect over *all* sandboxes allowing high -jN option to safely be used without worrying about overloading the system.
The specification is in https://www.gnu.org/software/make/manual/html_node/Job-Slots.html and client implementation PR for ninja is in https://github.com/ninja-build/ninja/pull/2506. The PR also contains Apache-licensed Python code that implements the server that can be used as inspiration.
Contributor guide
Research direction
Start with the GNU Make Job-Slots specification and the linked Ninja client PR, then inspect BuildStream's handling of --max-jobs and build sandboxes. Done means implementing the jobserver server, mounting its FIFO or pipe into sandboxes, and setting MAKEFLAGS so nested builds share the job limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100