Large actions with BwtB + dynamic execution never converge on local builds
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
The scenario is like this:
* We have some intermediate C++ libraries with lots of large inputs. Let's call one of these X.
* We enable BwtB to avoid transferring unnecessary intermediates in the common case.
* We enable dynamic execution for the whole build to optimize incremental builds like "rebuild single C++ file, relink the intermediate library X, run test".
When BwtB is disabled, dynamic execution in this scenario offers a massive speedup for step 3 and our developers really want to have quick turnaround under those types of C++ edits.
However, when we enable BwtB, the build of X never converges on local execution when the network is relatively slow. The problem goes like this:
* Bazel spawns the build of X locally and remotely.
* Bazel starts downloading the inputs of X, some of which are very large.
* The build of X finishes remotely.
* Bazel cancels the local build of X. This interrupts the download of the large inputs half-way, and those are deleted from disk.
When running this build multiple times, one would expect the chain of actions to happen purely locally at some point. But that's never the case: the build of X is always remote.
The problem here is triggered by the network being relatively slow: because some of the large inputs to X can never complete, X never has a chance to even start running locally. And thus even if running X locally would be faster overall, the remote build always finishes before Bazel has downloaded all inputs.
I think Bazel should either keep the partial downloads on disk and try to resume them later on a subsequent run, or continue the downloads in the background even if the local action is cancelled. I'm not sure what's preferable though. The former seems hard to implement and the latter can lead to problems in large builds with Bazel ending up with a long queue of downloads to process that may ultimately be useless...
### Which category does this issue belong to?
Local Execution
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
_No response_
### Which operating system are you running Bazel on?
N/A
### What is the output of `bazel info release`?
release 6.5.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
_No response_
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
_No response_
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
The report names BwtB, dynamic execution, and local/remote execution, but no source file, entry point, reproducer, or test. Start by locating Bazel's BwtB and dynamic-execution handling for cancelled local actions and input downloads, then reproduce the slow-network scenario. Done means repeated local builds converge as expected, with regression coverage for interrupted large-input downloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- build-system, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100