Best practices for parallelizing build tools and local execution

This bug tracks establishing best practices for optimizing Bazel build performance with build tools that support parallelism.

Some build tools can be optimized by parallelizing their work and using more cores. Actions can declare that they require additional CPUs with `exec_properties`, and after https://github.com/bazelbuild/bazel/pull/29284 the scheduler will respect those CPU reservations.

For build tools that can use additional cores to get a speedup, but that don't require additional CPUs, this won't always be an improvement for local execution. Reserving more CPUs means that fewer actions will be scheduled, and there will be less build graph level parallelism in the executed actions. Amdahl's law means that the speedup from parallelizing a single action is going to be sublinear with the number of cores, which is not as good as just running two actions in parallel.

The tradeoffs are different for remote execution if more parallelism is available.

It would be nice for actions to have a way to express that they can use more cores if they're available, and then for Bazel to decide how many cores to give the tool (potentially more for remove builds than local builds), and communicate that back to the tool. For remove execution, running the action in a container and setting cgroup limits is sufficient. For local actions that aren't containerized, this may require a new mechanism.

Start by reading PR #29284 and the scheduler behavior around exec_properties and CPU reservations. Compare the local and remote execution tradeoffs described here; done would require an agreed mechanism or design for optional CPU use by parallel-capable build tools.

Written by the indexing model from the issue text.