apache / apache/buildstream

feature: Support make jobserver protocol

Open
#1,968 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

Make has already for some time used jobserver protocol such that when a sub-make is run, top-make decides whether more processes can be spawned. This is a requirement for -jN option to work with nested makes. Recently, there's a PR to add jobserver *client* support to ninja as well. I propose buildstream to implement the actual jobserver part (at least fifo, can use also pipe) and mounts relevant fifo into build sandbox. There is also the part of setting MAKEFLAGS and declaring where the job server is. When this setup is fully in place and user passes --max-jobs to BuildStream, it can actually take effect over *all* sandboxes allowing high -jN option to safely be used without worrying about overloading the system.

The specification is in https://www.gnu.org/software/make/manual/html_node/Job-Slots.html and client implementation PR for ninja is in https://github.com/ninja-build/ninja/pull/2506. The PR also contains Apache-licensed Python code that implements the server that can be used as inspiration.

Contributor guide

Open the contributing guide

Research direction

Start with the GNU Make Job-Slots specification and the linked Ninja client PR, then inspect BuildStream's handling of --max-jobs and build sandboxes. Done means implementing the jobserver server, mounting its FIFO or pipe into sandboxes, and setting MAKEFLAGS so nested builds share the job limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.