facebook / facebook/buck2

Feature: act as Make job server when executing locally

Aperta
#653 0 commenti 7 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Rust
Stelle
4.4k
Fork
394
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

If you are building a lot of rust crates locally, they are all going to spin up at least a bunch of LLVM threads, and in future (or with nightly) they will [also spin up a bunch of threads](https://github.com/rust-lang/rust/issues/113349) for the rustc frontend. This is a bit of an issue because buck2 will saturate its own job queue's frontier (e.g. 8 concurrent builds with `-j8`) and then each of those `rustc` invocations will try to use as many threads as it can. You can often saturate N cores by passing `-j` with N/2 or so. Which means in the usual case of `-jN`, you are really scheduling about N^2 threads at once, which is surely not great.

There is an existing protocol for limiting the parallelism of individual build actions: the GNU Make jobserver: https://www.gnu.org/software/make/manual/html_node/Job-Slots.html. Rustc and I presume a number of other build tools are able to use the jobserver if you provide the environment variable (e.g. `MAKEFLAGS=--jobserver-auth=...`), and do the pipe IO / windows IPC serving. This is what tools like `cargo` do, and of course `make` itself. Ultimately all the participating processes share a single semaphore.

See [here](https://github.com/rust-lang/rust/blob/587af910459fe408f03e004d264fdf218203849d/compiler/rustc_interface/src/util.rs#L123) for where rustc acquires and releases jobserver tokens.

The jobserver crate has a type `Client` that also acts as a server. If you call `Client::new` when starting a build, pass in the buck2 CLI's `-j` flag, and use its methods to attach the environment variables to local execution processes, you are basically done.

https://docs.rs/jobserver/latest/jobserver/struct.Client.html#method.new

I am sure Meta will not care about this, given you scale your builds horizontally as parallel as you like, but it would be pretty helpful for OSS builds.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.