Bazel parallelism

Some of our Bazel jobs are significantly oversubscribing the available hardware threads during builds, slowing them down and making it harder to do anything else (e.g. edit files) in parallel.

An example is the Rust builds. Bazel spawns nproc Rust compilers which each spawn (up to) nproc parallel threads. Normally these cooperate using the jobserver protocol which Bazel does not use.

Limiting rustc to one codegen unit/thread gives a small compile time improvement and (more importantly) frees up the OS scheduler slightly to let you do other things:

hyperfine --prepare 'bazel clean && bazel build //sw/device/lib/testing/json/...' --runs 10 'bazel build //sw/host/opentitantool'
hyperfine --prepare 'bazel clean && bazel build //sw/device/lib/testing/json/...' --runs 10 'bazel build //sw/host/opentitantool --@rules_rust//rust/settings:codegen_units=1'

Benchmark 1: bazel build //sw/host/opentitantool
  Time (mean ± σ):     138.998 s ±  1.491 s    [User: 0.317 s, System: 0.276 s]
  Range (min … max):   136.610 s … 141.255 s    10 runs

Benchmark 2: bazel build //sw/host/opentitantool --@rules_rust//rust/settings:codegen_units=1
  Time (mean ± σ):     123.210 s ±  1.705 s    [User: 0.256 s, System: 0.201 s]
  Range (min … max):   120.531 s … 125.392 s    10 runs

Still slower than Cargo (in theory) but an improvement.

We'll probably have the same issue with LTO soon and maybe some make builds (Verilator?).

Should we consider limiting per-process concurrency and only allowing Bazel to control concurrency through creating more processes?

Start by reproducing the issue with the two hyperfine Bazel commands in the report, then inspect the Bazel and rules_rust concurrency settings, including codegen_units=1. Compare build time and system responsiveness, and define whether the desired result is a documented or implemented limit for per-process concurrency.