facebook / facebook/buck2

Feature: act as Make job server when executing locally

Ouverte
#653 0 commentaires 7 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
Rust
Étoiles
4.4k
Forks
394
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

If you are building a lot of rust crates locally, they are all going to spin up at least a bunch of LLVM threads, and in future (or with nightly) they will [also spin up a bunch of threads](https://github.com/rust-lang/rust/issues/113349) for the rustc frontend. This is a bit of an issue because buck2 will saturate its own job queue's frontier (e.g. 8 concurrent builds with `-j8`) and then each of those `rustc` invocations will try to use as many threads as it can. You can often saturate N cores by passing `-j` with N/2 or so. Which means in the usual case of `-jN`, you are really scheduling about N^2 threads at once, which is surely not great.

There is an existing protocol for limiting the parallelism of individual build actions: the GNU Make jobserver: https://www.gnu.org/software/make/manual/html_node/Job-Slots.html. Rustc and I presume a number of other build tools are able to use the jobserver if you provide the environment variable (e.g. `MAKEFLAGS=--jobserver-auth=...`), and do the pipe IO / windows IPC serving. This is what tools like `cargo` do, and of course `make` itself. Ultimately all the participating processes share a single semaphore.

See [here](https://github.com/rust-lang/rust/blob/587af910459fe408f03e004d264fdf218203849d/compiler/rustc_interface/src/util.rs#L123) for where rustc acquires and releases jobserver tokens.

The jobserver crate has a type `Client` that also acts as a server. If you call `Client::new` when starting a build, pass in the buck2 CLI's `-j` flag, and use its methods to attach the environment variables to local execution processes, you are basically done.

https://docs.rs/jobserver/latest/jobserver/struct.Client.html#method.new

I am sure Meta will not care about this, given you scale your builds horizontally as parallel as you like, but it would be pretty helpful for OSS builds.

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Start with the jobserver crate's Client::new documentation and the linked rustc_interface/src/util.rs section to understand token acquisition. Trace Buck2's local execution path and its -j handling, then attach the jobserver environment to local processes. Done means local rustc and other participating tools share the configured parallelism limit.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
rust
Domaine
build-system
Type d'issue
Fonctionnalité
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
45/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.