Feature: act as Make job server when executing locally
- Lingua principale
- Rust
- Stelle
- 4.4k
- Fork
- 394
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
If you are building a lot of rust crates locally, they are all going to spin up at least a bunch of LLVM threads, and in future (or with nightly) they will [also spin up a bunch of threads](https://github.com/rust-lang/rust/issues/113349) for the rustc frontend. This is a bit of an issue because buck2 will saturate its own job queue's frontier (e.g. 8 concurrent builds with `-j8`) and then each of those `rustc` invocations will try to use as many threads as it can. You can often saturate N cores by passing `-j` with N/2 or so. Which means in the usual case of `-jN`, you are really scheduling about N^2 threads at once, which is surely not great.
There is an existing protocol for limiting the parallelism of individual build actions: the GNU Make jobserver: https://www.gnu.org/software/make/manual/html_node/Job-Slots.html. Rustc and I presume a number of other build tools are able to use the jobserver if you provide the environment variable (e.g. `MAKEFLAGS=--jobserver-auth=...`), and do the pipe IO / windows IPC serving. This is what tools like `cargo` do, and of course `make` itself. Ultimately all the participating processes share a single semaphore.
See [here](https://github.com/rust-lang/rust/blob/587af910459fe408f03e004d264fdf218203849d/compiler/rustc_interface/src/util.rs#L123) for where rustc acquires and releases jobserver tokens.
The jobserver crate has a type `Client` that also acts as a server. If you call `Client::new` when starting a build, pass in the buck2 CLI's `-j` flag, and use its methods to attach the environment variables to local execution processes, you are basically done.
https://docs.rs/jobserver/latest/jobserver/struct.Client.html#method.new
I am sure Meta will not care about this, given you scale your builds horizontally as parallel as you like, but it would be pretty helpful for OSS builds.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.