Rethink spawn strategies
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
We've seen a lot of reported issues that, one way or another, reflect a lack of flexibility in the way we apply spawn strategies.
We believe it's time to rethink how spawn strategies are implemented and what the right API to choose among them should look like. In particular, it would be nice to:
- Reimplement the disk/remote cache lookup as a strategy (instead of as part of the spawn runner)
- Generalize dynamic execution to an arbitrary number of strategies (or combinations of strategies) to be tried in parallel
- Generalize local fallback to an arbitrary number of strategies (or combinations of strategies) to be tried in series
- Have a way to distinguish "permanent" from "transient" failures (permanent are not worth retrying with a different strategy, transient might be)
- Make it possible to toggle strategies along dimensions other than mnemonic (e.g. based on the execution platform, selected toolchain, or estimated resource consumption)
This would make it possible to provide solutions to the following issues:
- [ ] https://github.com/bazelbuild/bazel/issues/7202
- [ ] https://github.com/bazelbuild/bazel/issues/7328
- [ ] https://github.com/bazelbuild/bazel/issues/7664
- [ ] https://github.com/bazelbuild/bazel/issues/11432
- [ ] https://github.com/bazelbuild/bazel/issues/15519
- [ ] https://github.com/bazelbuild/bazel/issues/16132
- [x] https://github.com/bazelbuild/bazel/issues/18140
- [x] https://github.com/bazelbuild/bazel/issues/18244
- [ ] https://github.com/bazelbuild/bazel/issues/18245
- [x] https://github.com/bazelbuild/bazel/issues/18319
- [ ] https://github.com/bazelbuild/bazel/issues/20123
- [x] https://github.com/bazelbuild/bazel/issues/21725
In particular, #11432 references https://docs.google.com/document/d/1U9HzdDmtRnm244CaRM6JV-q2408mbNODAMewcGjnnbM/edit#heading=h.5mcn15i0e1ch which contains many interesting ideas we should revisit in light of this set of requirements.
Contributor guide
Research direction
Start by reviewing the linked issues, especially #11432, and the referenced Google design document to understand the existing spawn-strategy requirements. This issue is complete only when the strategy model and selection API are agreed and the listed use cases can be addressed; no source files or tests are identified here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100