bazelbuild / bazelbuild/remote-apis
Protect against partial actions in the cache
- Dominant language
- Go
- Stars
- 445
- Forks
- 141
- PR merge metrics
- No merged PRs in 30d
Description
Currently, when an execution fails, we allow returning a partially-populated `ActionResult` message. RBE experienced a failure where these partial results were making it into the cache, and because the `exit_code` field was unpopulated, it was interpreted as 0. Subsequent builds read the result from the cache and interpreted it as successful, but since it had no output files the build failed later when the requisite file was not present. It's certainly believable that a failure like this could reoccur in RBE or in other implementations, so we would like to put protections in place against it.
As I understand it, Bazel's architecture does easily lend itself to having outputs be mandatory, which is why it can only detect the failure downstream. This is why all outputs are considered optional at the API level; even trying to separate out optional and mandatory outputs on the Bazel side might prove difficult.
One suggestion was to require that all action results have at least one output file or directory to be considered valid; an action that has no meaningful output files could add a dummy output and touch it on the bot side (or even include it as an input) to ensure that the empty `ActionResult` is not propagated.
Contributor guide
Research direction
Start by reviewing how ActionResult messages are validated before entering the cache, with particular attention to exit_code and output files or directories. Trace the failure path described in the issue and determine what validity rule prevents partial results from being propagated. Done means failed or empty results cannot be interpreted as successful cached actions, including actions without meaningful outputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100