Cooperative Parallelism

### Description of the problem / feature request:

This is an umbrella issue of problems that arise from using build tools that have their own internal parallelism.

In this Google Groups thread, @jmmv asked to file an issue about this:
> https://groups.google.com/d/msg/bazel-discuss/_oHaU50P5Rg/imx5Y49MAwAJ

A little context: `swiftc` is the swift compiler driver. It's a non-traditional compiler, it doesn't build one source file at time, it builds one module of N source files at time. `swiftc` spawns "swift frontend" invocations, and the number of spawned processes is very often >1.

There are two related problems:

1. Tools that perform parallel sub-actions cannot express this use of parallelism to bazel
2. Actions have no API through which they can specify maximum parallelism

In the first case, it would be good if the action API could express to Bazel how much parallelism is used by an action. This avoids the problem of N bazel actions each running some M sub-actions each.

In the second case, it would be good if the action API could express a range of parallelism an action is capable of using. This would really help the performance of bottleneck actions in the critical path. For example, Bazel could see that it's not using its full amount of jobs, and donate the extra parallelism to the bottleneck action. We see this as particularly useful at the tail end of builds, where there are fewer targets left to build. This problem shows up even more in incremental builds, where the action graph is often much more flat, even linear.

As @allevato pointed out in the google groups thread, this would require some way for actions to pass args that are known not to affect output, such as a `-j` flag. This would also need to preserve the cache keys.

### Feature requests: what underlying problem are you trying to solve with this feature?

This feature allows us to avoid two current problems:

1. prevent oversubscribing cpu during the build
2. prevent slower-than-necessary builds that are caused by wasted parallelism

The first issue can happen with any swift module over 25 files. The default batching logic creates one swift frontend for each group of 25 files. A swift module with 100 files will spawn 4 sub-actions, unbeknownst to Bazel.

As mentioned, the second case is something that causes slowdowns for incremental development builds.

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

If needed, I can make a rules_swift project that demonstrates the issue.

We see the problem in our build by looking at `--experimental_generate_json_trace_profile` and by comparing to Xcode's builds, which can sometimes be faster due to its seemingly hard code use of `-j8`.

### What operating system are you running Bazel on?

macOS

### What's the output of `bazel info release`?

release 1.2.0

### Have you found anything relevant by searching the web?

As mentioned above, a small amount of discussion happened on Google Groups:

> https://groups.google.com/d/msg/bazel-discuss/_oHaU50P5Rg/imx5Y49MAwAJ

I've also posted a general (non-bazel) question to the Swift Forums.

> https://forums.swift.org/t/globally-optimized-build-parallelism/31802/2

Start with the linked Google Groups discussion and the issue's description of swiftc spawning multiple frontend processes. Compare the two proposed action APIs: reporting internal parallelism and declaring a maximum or range. Done would require a concrete Bazel design that avoids CPU oversubscription, preserves cache keys, and improves available parallelism without an implementation location or test being identified here.

Written by the indexing model from the issue text.