bazelbuild / bazelbuild/remote-apis

Protect against partial actions in the cache

Open
#6 24 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
445
Forks
141
PR merge metrics
No merged PRs in 30d

Description

Currently, when an execution fails, we allow returning a partially-populated `ActionResult` message. RBE experienced a failure where these partial results were making it into the cache, and because the `exit_code` field was unpopulated, it was interpreted as 0. Subsequent builds read the result from the cache and interpreted it as successful, but since it had no output files the build failed later when the requisite file was not present. It's certainly believable that a failure like this could reoccur in RBE or in other implementations, so we would like to put protections in place against it.

As I understand it, Bazel's architecture does easily lend itself to having outputs be mandatory, which is why it can only detect the failure downstream. This is why all outputs are considered optional at the API level; even trying to separate out optional and mandatory outputs on the Bazel side might prove difficult.

One suggestion was to require that all action results have at least one output file or directory to be considered valid; an action that has no meaningful output files could add a dummy output and touch it on the bot side (or even include it as an input) to ensure that the empty `ActionResult` is not propagated.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing how ActionResult messages are validated before entering the cache, with particular attention to exit_code and output files or directories. Trace the failure path described in the issue and determine what validity rule prevents partial results from being propagated. Done means failed or empty results cannot be interpreted as successful cached actions, including actions without meaningful outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.