bazelbuild / bazelbuild/bazel

Rethink spawn strategies

Open
#19,904 3 comments 10 reactions 0 assignees View on GitHub
team-Remote-Exec type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

We've seen a lot of reported issues that, one way or another, reflect a lack of flexibility in the way we apply spawn strategies.

We believe it's time to rethink how spawn strategies are implemented and what the right API to choose among them should look like. In particular, it would be nice to:

- Reimplement the disk/remote cache lookup as a strategy (instead of as part of the spawn runner)
- Generalize dynamic execution to an arbitrary number of strategies (or combinations of strategies) to be tried in parallel
- Generalize local fallback to an arbitrary number of strategies (or combinations of strategies) to be tried in series
- Have a way to distinguish "permanent" from "transient" failures (permanent are not worth retrying with a different strategy, transient might be)
- Make it possible to toggle strategies along dimensions other than mnemonic (e.g. based on the execution platform, selected toolchain, or estimated resource consumption)

This would make it possible to provide solutions to the following issues:

- [ ] https://github.com/bazelbuild/bazel/issues/7202
- [ ] https://github.com/bazelbuild/bazel/issues/7328
- [ ] https://github.com/bazelbuild/bazel/issues/7664
- [ ] https://github.com/bazelbuild/bazel/issues/11432
- [ ] https://github.com/bazelbuild/bazel/issues/15519
- [ ] https://github.com/bazelbuild/bazel/issues/16132
- [x] https://github.com/bazelbuild/bazel/issues/18140
- [x] https://github.com/bazelbuild/bazel/issues/18244
- [ ] https://github.com/bazelbuild/bazel/issues/18245
- [x] https://github.com/bazelbuild/bazel/issues/18319
- [ ] https://github.com/bazelbuild/bazel/issues/20123
- [x] https://github.com/bazelbuild/bazel/issues/21725

In particular, #11432 references https://docs.google.com/document/d/1U9HzdDmtRnm244CaRM6JV-q2408mbNODAMewcGjnnbM/edit#heading=h.5mcn15i0e1ch which contains many interesting ideas we should revisit in light of this set of requirements.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked issues, especially #11432, and the referenced Google design document to understand the existing spawn-strategy requirements. This issue is complete only when the strategy model and selection API are agreed and the listed use cases can be addressed; no source files or tests are identified here.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.