Feature: act as Make job server when executing locally
- Vorherrschende Sprache
- Rust
- Sterne
- 4.4k
- Forks
- 394
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
If you are building a lot of rust crates locally, they are all going to spin up at least a bunch of LLVM threads, and in future (or with nightly) they will [also spin up a bunch of threads](https://github.com/rust-lang/rust/issues/113349) for the rustc frontend. This is a bit of an issue because buck2 will saturate its own job queue's frontier (e.g. 8 concurrent builds with `-j8`) and then each of those `rustc` invocations will try to use as many threads as it can. You can often saturate N cores by passing `-j` with N/2 or so. Which means in the usual case of `-jN`, you are really scheduling about N^2 threads at once, which is surely not great.
There is an existing protocol for limiting the parallelism of individual build actions: the GNU Make jobserver: https://www.gnu.org/software/make/manual/html_node/Job-Slots.html. Rustc and I presume a number of other build tools are able to use the jobserver if you provide the environment variable (e.g. `MAKEFLAGS=--jobserver-auth=...`), and do the pipe IO / windows IPC serving. This is what tools like `cargo` do, and of course `make` itself. Ultimately all the participating processes share a single semaphore.
See [here](https://github.com/rust-lang/rust/blob/587af910459fe408f03e004d264fdf218203849d/compiler/rustc_interface/src/util.rs#L123) for where rustc acquires and releases jobserver tokens.
The jobserver crate has a type `Client` that also acts as a server. If you call `Client::new` when starting a build, pass in the buck2 CLI's `-j` flag, and use its methods to attach the environment variables to local execution processes, you are basically done.
https://docs.rs/jobserver/latest/jobserver/struct.Client.html#method.new
I am sure Meta will not care about this, given you scale your builds horizontally as parallel as you like, but it would be pretty helpful for OSS builds.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.