Autoscaling '--jobs' during a build/test depending on the available memory?

### feature request:

Autoscaling `--jobs` during a build/test depending on the available memory. Specifically, if running with `--jobs` higher than one and a build step fails due to `virtual memory exhausted: Cannot allocate memory`, then scale down the actual number of parallel jobs and automatically repeat the failed build step. Repeat this until the actual number of parallel jobs gets down to one and the compilation still fails, and fail the build only if it still fails with just one parallel job.

### Feature requests: what underlying problem are you trying to solve with this feature?

This would solve practical problem with automated builds, where we currently have had to permanently cut `--jobs` to just one due to a small fraction of the source files requiring huge amount of memory to build. This has then caused the project build to take upto 1.5 hours often leading to build cancellation due to the build system timeout. If build would automatically scale between 1 and '--jobs' (e.g., 4) parallel builds as allowed by available memory, the aforementioned build would take less than half an hour.

On local builds this feature would remove the need to manually restart build with lower '--jobs' count after compiler out-of-memory errors, and then quitting and manually restarting with higher '--jobs' count after passing the "memory bottleneck" to speed up compilation again.

### What operating system are you running Bazel on?

Linux

### What's the output of `bazel info release`?

```
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
/home/vagrant/dev/envoyproxy/envoy/tools/bazel.rc
release 1.1.0
```

### If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.

N/A

### What's the output of `git remote get-url origin ; git rev-parse master ; git rev-parse HEAD` ?

```
https://github.com/jrajahalme/envoy.git
7f8fb9509d3189819dd253e25ec76e939ae106e7
0a9519362f1c6e5646e9f5843cd254451be183a0
```

### Have you found anything relevant by searching the web?

No

### Any other information, logs, or outputs that you want to share?

Out of memory failures typically happen when running multiple jobs and build proceeding to gmock/gtest C++ source files. Non-gmock/gtest source files are happily building fast with two or three parallel jobs on the same VM.

Example output for failing with two jobs, then succeeding with one:

```
$ bazel test //test/server:server_fuzz_test --jobs=2 --test_arg="-l debug"
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
/home/vagrant/dev/envoyproxy/envoy/tools/bazel.rc
INFO: Writing tracer profile to '/home/vagrant/.cache/bazel/_bazel_vagrant/3cd2b6284cb460c61be41e0c1b844cea/command.profile.gz'
INFO: Analyzed target //test/server:server_fuzz_test (1 packages loaded, 364 targets configured).
INFO: Found 1 test target...
ERROR: /home/vagrant/dev/envoyproxy/envoy/test/mocks/upstream/BUILD:62:1: C++ compilation of rule '//test/mocks/upstream:upstream_mocks' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 485 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
virtual memory exhausted: Cannot allocate memory
Target //test/server:server_fuzz_test failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 240.349s, Critical Path: 75.69s
INFO: 11 processes: 11 linux-sandbox.
FAILED: Build did NOT complete successfully

FAILED: Build did NOT complete successfully

$ bazel test //test/server:server_fuzz_test --jobs=1 --test_arg="-l debug"
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
/home/vagrant/dev/envoyproxy/envoy/tools/bazel.rc
INFO: Writing tracer profile to '/home/vagrant/.cache/bazel/_bazel_vagrant/3cd2b6284cb460c61be41e0c1b844cea/command.profile.gz'
INFO: Analyzed target //test/server:server_fuzz_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //test/server:server_fuzz_test up-to-date:
bazel-bin/test/server/server_fuzz_test
INFO: Elapsed time: 290.465s, Critical Path: 92.88s
INFO: 8 processes: 8 linux-sandbox.
INFO: Build completed successfully, 9 total actions
//test/server:server_fuzz_test PASSED in 1.9s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 9 total actions
```

Start by reproducing the issue with `bazel test //test/server:server_fuzz_test --jobs=2` and compare it with `--jobs=1`. The reported failure occurs while compiling `test/mocks/upstream:upstream_mocks` from `test/mocks/upstream/BUILD:62`; investigate Bazel's scheduling and retry entry points. Done means memory-related failures reduce parallelism and retry automatically, while a failure at one job still fails the build.

Written by the indexing model from the issue text.