bazelbuild / bazelbuild/bazel

Support non-speculative hybrid local/remote execution (work sharing across distinct actions)

Open
#31,094 0 comments 0 reactions 0 assignees View on GitHub
team-Local-Exec team-Remote-Exec type: feature request untriaged
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

I would like Bazel to support a non-speculative hybrid local/remote execution mode in which local CPU resources and remote execution resources form two cooperating execution pools and execute **distinct actions**.

The desired behavior is different from the existing `dynamic` spawn strategy.

For example, suppose:

* the remote executor can effectively execute 76 actions concurrently;
* the developer workstation has 10 CPUs available for local execution;
* the build contains many independent actions that are eligible for both local and remote execution.

I would like Bazel to be able to execute approximately 86 distinct actions concurrently:

```text
76 distinct actions executing remotely
10 distinct actions executing locally
---------------------------------------
86 distinct actions making progress
```

The scheduler should assign each ready action to either the local or remote pool, depending on available capacity. An individual action should normally be executed only once.

Conceptually:

```text
ready actions
|
+----------+----------+
| |
local capacity remote capacity
10 CPUs ~76 actions
| |
distinct actions distinct actions
```

This is intentionally different from:

```text
--spawn_strategy=dynamic
```

Dynamic execution uses speculative execution: the same action can have a local and a remote branch, and whichever branch finishes first wins while the other is cancelled.

That is useful for minimizing the latency of an individual action, but it does not aggregate local and remote compute capacity. If 10 actions are racing both locally and remotely, those 10 local CPUs are duplicating work already being attempted remotely instead of executing 10 additional actions.

What I am requesting is work sharing rather than racing/speculation.

A possible scheduler invariant would be:

```text
For every runnable action that is eligible for both local and remote execution:

- execute the action exactly once under normal operation;
- if a local execution slot is available, Bazel may assign the action locally;
- otherwise Bazel may assign it remotely;
- keep both local and remote pools busy while independent runnable actions exist.

Local execution remains constrained by --local_resources.
Remote concurrency can be governed separately from local resources.
```

I am not proposing a specific command-line interface. This could be a new spawn strategy, an option modifying dynamic execution, or a more general execution scheduler change.

For illustration only, something conceptually equivalent to this would be sufficient:

```text
local capacity: cpu=10
remote capacity: 76 concurrent actions

scheduler: use both pools for distinct work
```

It is important that users do not have to statically decide which mnemonics, targets, packages, or actions execute locally versus remotely. The point of the feature is for Bazel to make that decision dynamically based on currently available execution capacity.

It should therefore work when the same action is valid on either execution environment and there is no semantic reason to prefer one over the other.

### Which category does this issue belong to?

Local Execution, Remote Execution, Performance

### What underlying problem are you trying to solve with this feature?

For developer builds using remote execution, the developer's workstation often has significant CPU capacity sitting idle. This is especially the case for macOS where RBE systems usually lack significant compute power, while developers' macOS workstations sit idle.

Consider a developer with 10 usable local CPU cores connected to a remote execution service that provides approximately 76 concurrent execution slots.

With remote execution, Bazel can keep the remote workers busy, but the developer's 10 local CPUs do not contribute additional throughput for actions that could have executed either locally or remotely.

Using:

```text
--spawn_strategy=dynamic
```

does use the local CPUs, but it uses them to race copies of actions already executing remotely. This trades additional compute consumption for lower latency of individual actions.

That is not what I want.

For a sufficiently parallel build, executing:

```text
action A remotely
action B remotely
...
action X remotely

action Y locally
action Z locally
...
```

should generally provide more aggregate throughput than executing:

```text
action A remotely + another copy of action A locally
action B remotely + another copy of action B locally
...
```

when the remote executor already has substantial capacity.

The desired behavior is analogous to adding the workstation's execution capacity to the remotely available capacity for the duration of that developer's build, without actually registering the workstation as a shared remote worker.

Registering the workstation with the remote execution cluster is not an equivalent solution:

1. In many environments developers cannot add arbitrary workers to the remote execution pool.
2. Even where that is technically possible, a developer workstation should generally remain private to that developer's invocation rather than receiving actions from unrelated developers.
3. The workstation is already directly controlled by the Bazel client, so routing its capacity through the shared remote scheduler should not be necessary merely to use it as additional build capacity.

Static strategy configuration is also not an equivalent solution. For example, manually assigning one mnemonic to `local` and another to `remote`, or using `--strategy_regexp`, requires knowing in advance how to partition the action graph.

The optimal partition can change between builds and over the course of a single build. The scheduler already knows which actions are ready and whether local resources are occupied, so this seems like a scheduling decision rather than something the user should have to encode statically.

The practical goal is:

```text
remote workers busy
AND
local CPUs busy
AND
no intentional duplicate execution
```

for as long as enough independent runnable actions exist.

### Which operating system are you running Bazel on?

Platform-independent. This is an execution scheduling feature request rather than an OS-specific issue.

### What is the output of `bazel info release`?

N/A — this feature request is not specific to one Bazel release. The relevant distinction is between Bazel's current priority-based spawn-strategy selection and speculative `dynamic` execution, and the requested non-speculative work-sharing behavior.

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

N/A

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text
N/A
```

### Have you found anything relevant by searching the web?

I found several related Bazel issues, but none that appear to request the same behavior: using local and remote execution as two non-speculative pools that process different ready actions concurrently.

Relevant issues:

* https://github.com/bazelbuild/bazel/issues/7327 — **Remove "local execution delay" from dynamic spawn scheduler**

This discusses when the dynamic scheduler should start the local copy of an action that is already being attempted remotely. One concern raised in the discussion is that starting local work too aggressively can schedule useless duplicate actions, consume local resources, and delay actions that genuinely need to run locally.

The proposal here is different: local resources would execute other ready actions rather than duplicate an action already running remotely. In that sense, it may avoid some of the resource-waste concerns discussed in #7327.

* https://github.com/bazelbuild/bazel/issues/7345 — **"X actions, Y running" is misleading with the dynamic spawn scheduler**

This describes how dynamic execution has two strategies active for the same action: a remote branch and a local branch. It also illustrates that local branches may be waiting for local resources while many remote actions are already running.

This issue is about progress reporting rather than scheduling policy, but it highlights the distinction between today's speculative dynamic execution and the proposed work-sharing behavior.

* https://github.com/bazelbuild/bazel/issues/7818 — **Make the dynamic strategy work without sandboxing**

This discusses technical complications caused by racing local and remote copies of the same action, including output coordination, cancellation, and sandboxing.

Those problems mostly arise from speculative duplication. The requested behavior here would instead assign different actions to local and remote execution, so both pools contribute useful work without racing to produce the same outputs.

* https://github.com/bazelbuild/bazel/issues/7932 — **Local actions in a remote execution build should be cachable**

This explicitly discusses hybrid builds involving both local and remote execution. There was initially concern about mixing locally executed results with remote caching because of cache-poisoning risks, but that concern was subsequently clarified and the relevant functionality was implemented.

This issue does not address dynamically distributing otherwise-remotable actions between available local and remote execution capacity.

* https://github.com/bazelbuild/bazel/issues/3875 — **bazel, distcc and remote execution?**

This is an older discussion involving high remote compilation parallelism together with local actions and local resource accounting. Bazel maintainers indicated that native remote execution, rather than distcc, was the intended long-term solution.

It does not appear to discuss combining native remote execution capacity with otherwise-idle local CPU capacity by assigning different actions to each.

* https://github.com/bazelbuild/bazel/issues/25793 — **Resource use for local actions should be able to be specified via exec_properties (and exec groups)**

This concerns consistent resource sizing and resource hints for local and remote execution. It addresses how much resource an action should consume, rather than how Bazel should choose between available local and remote execution capacity.

I also searched closed issues specifically to see whether this kind of non-speculative local/remote work sharing had previously been proposed and rejected. I did not find such an issue or a maintainer decision against the model.

The closest existing mechanism still appears to be dynamic execution, but dynamic execution races local and remote copies of the same action. The feature requested here is different: treat local and remote execution as complementary capacity and schedule distinct ready actions onto whichever pool has capacity, without intentionally duplicating work.

### Any other information, logs, or outputs that you want to share?

A few properties that I think are important for the intended semantics:

1. **No static action partitioning**

Users should not need to predict which targets, mnemonics, packages, or individual actions should run locally.

2. **No duplicate execution under normal operation**

If an action has been assigned to the remote pool, the local scheduler should preferably take another ready action rather than starting a second copy of the same action.

This distinguishes the feature from dynamic/speculative execution.

3. **Separate resource limits**

Local execution should continue to respect Bazel's local CPU/RAM resource accounting, e.g. `--local_resources`.

Remote execution should have an independent concurrency mechanism rather than competing with local actions for the same CPU resource accounting.

4. **Capacity-aware scheduling**

If all local slots are occupied, additional actions can go remote.

If a local slot becomes free while there are ready actions, Bazel should be able to assign one of those ready actions locally instead of leaving the CPU idle.

5. **No requirement to expose the workstation as a shared remote worker**

The local execution capacity belongs only to the current Bazel invocation.

6. **Actions must still obey execution compatibility**

This proposal only concerns actions for which both execution methods are valid. Actions that are `no-remote`, `no-local`, require a particular execution platform, etc. should continue to obey those constraints.

7. **Remote cache semantics should remain normal**

A local action should still be able to benefit from an applicable remote cache entry, and locally produced deterministic results should follow the normal configured cache-upload policy.

The exact implementation/API is less important to me than the scheduling semantics.

A concise description of the requested behavior would be:

> Allow Bazel to use local execution resources as additional capacity alongside remote execution, automatically distributing distinct ready actions between the local and remote pools instead of speculatively executing the same action in both places.

For the example above, the desired steady-state behavior is approximately 76 remote actions + 10 local actions = 86 distinct actions executing concurrently, subject to dependencies and resource constraints.

Contributor guide

Open the contributing guide

Research direction

Start by reading Bazel's existing dynamic spawn strategy and local resource accounting, especially the behavior described for --local_resources, then compare the related issues linked in the report. Done means a scheduler design and implementation can assign distinct ready actions to local and remote pools, keep both pools productive, and avoid intentional duplicate execution under normal operation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.