Build being very slow when cross-compiling for macOS using RBE, might due to Mac machine being very powerless

### Description of the bug:

Hello,

We (ML DevInfra team at Google) have been trying to set up a build for TensorFlow that can cross-compile for macOS x86 on remote Linux x86 VMs using RBE. The build currently runs into issues if we set `--jobs` to be > 100. It seems to timeout at the final 2 actions. It is hard to tell what is happening from the description of the action ("checking cached actions") and running with `-s` does not seem to give any additional details. Can anyone on the Bazel side help us understand why it times out at the final 2 actions and if/how it can be fixed?

```
[45,720 / 45,736] [Sched] Linking tensorflow/python/_pywrap_tfcompile.so [for tool]; 7s ... (4 actions, 0 running)
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions
[45,734 / 45,736] checking cached actions

ERROR: Aborting VM command due to timeout of 43200 seconds
```

### Which category does this issue belong to?

Remote Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Please note that we have only ran into this issue in our CI build and running with our RBE setup would require additional permissions so this might be hard to reproduce but here are the commands to run nonetheless

```
1. git clone https://github.com/tensorflow/tensorflow.git && cd tensorflow
# Replace "test:" cross-compile configs with "build:"
2. sed -i '' 's/test:macos_x86_pycpp_test_filters/build:macos_x86_pycpp_test_filters/g' .bazelrc && sed -i '' 's/test:macos_x86_pycpp_test/build:macos_x86_pycpp_test/g' .bazelrc && sed -i '' 's/test:cross_compile_macos_x86_pycpp_test/build:cross_compile_macos_x86_pycpp_test/g' .bazelrc
3. `=bazel build --repo_env=TF_PYTHON_VERSION=3.11 --noremote_accept_cached --noenable_runfiles --jobs=300 --config rbe_cross_compile_macos_x86 --config=cross_compile_macos_x86_pycpp_test
```

### Which operating system are you running Bazel on?

macOS Ventura

### What is the output of `bazel info release`?

release 6.5.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

_No response_

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

We have a cross-compile build running for Linux Arm64 as well where we have seen this action appear in the log but that build does not get stuck indefinitely. The Mac machines we have in our CI are not powerful (for instance, they have 16GB RAM compared to 192 GB on our Linux Arm64 VMs) so our current suspicion is that the host Mac machine might be too weak to support remote building on > 100 machines.

Specs of the Mac machines in our CI:
OS | macOS 13.6 22G120
Proc Type | Intel(R) Core(TM) i7-4578U CPU @ 3.00GHz
Proc Desc. | Intel x86_64 Family 6 Model 85 Stepping 4
Proc Cores | 2
Proc Logical CPUs | 4
Total Memory (GB) | 16.00