bazelbuild / bazelbuild/bazel

Allow downstream remote-cache hits without materializing uncached producer outputs

Open
#30,698 3 comments 0 reactions 0 assignees View on GitHub
team-Remote-Exec type: bug untriaged
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

## Problem

We have a large monorepo that uses remote caching without remote execution, together with build-without-the-bytes (`--remote_download_minimal`).

Many Go tests depend on expensive `GoLink` actions. We intentionally do not cache the normal `GoLink` outputs because:

- the linked test binaries are very large and expensive to upload;
- link inputs and outputs churn frequently enough that uploading them is often not economical; and
- when the downstream test is already cached, the linked binary bytes are not needed at all.

Today Bazel must still execute `GoLink` locally before it can determine that the downstream test is a remote-cache hit:

1. `GoLink` has no usable remote action result.
2. Bazel executes `GoLink` and produces the test binary.
3. Bazel can then compute and query the test action.
4. The test is found in the remote cache, so the newly linked binary is never used.

At monorepo scale, this means many expensive links can execute solely to discover that their downstream tests were cached.

The general problem is that Bazel cannot query a downstream action until the metadata for all of its inputs is available, even when the downstream cache hit would make some producer output bytes unnecessary.

## Desired behavior

When sufficient producer identity or output metadata is available, Bazel should be able to determine whether a downstream action is cached before materializing an expensive producer output.

The desired behavior is:

- On a downstream cache hit, skip both the producer and downstream execution.
- On a downstream cache miss, execute the producer and downstream action normally.
- Preserve normal invalidation when the producer definition, inputs, test definition, or other downstream inputs change.
- Avoid uploading the producer's large output bytes.
- Work with a standard REAPI remote cache without requiring server changes.
- Fail open to normal execution if the optimization cannot be used.

Although Go tests are our motivating case, we would prefer an abstraction that can apply to other producer/consumer action pairs where appropriate.

## Implementation proposals

We have explored two Bazel-core prototypes. These are intended to make the design space concrete, not to prescribe the final solution.

### 1. Metadata-only producer action results

Draft PR: https://github.com/bazelbuild/bazel/pull/30635

This approach stores producer output metadata under a separate derived action key while omitting the producer's output blobs.

On a later build, Bazel restores that metadata so it can compute downstream action keys without executing the producer. If a downstream cache miss actually requires the absent bytes, Bazel falls back to executing the producer through lost-input recovery.

Advantages:

- Generalizes naturally to producer actions beyond tests.
- Reuses Bazel's normal downstream action-key calculation.
- Represents the real producer outputs rather than introducing a separate downstream identity.

Tradeoffs and open questions:

- ~~Correct fallback depends on missing-input recovery/action-rewinding. Which when tested in our monorpeo, added much more latency/churn than the normal cache-miss path.~~ Confirmed a non-issue on Bazel 8.7+
- Bazel temporarily treats output bytes as available when only their metadata exists.
- At large scale, missing bytes may be discovered by different downstream consumers at different times.
- It is unclear whether metadata-only results fit naturally into Bazel's existing action-cache model.

### 2. Producer-keyed test-cache lookup

Draft PR: https://github.com/bazelbuild/bazel/pull/30636

This approach computes an alternate test-cache key from:

- the producer action definition and inputs;
- the test definition; and
- the test's other inputs and runfiles.

Bazel queries this key before requesting the generated test executable. On a hit, it restores the completed test result immediately. On a miss, normal `GoLink` and test execution continue.

Advantages:

- A miss follows the normal execution path without introducing missing output bytes.
- Does not require storing a partial producer result in the remote cache.
- The prototype has successfully skipped both `GoLink` and `TestRunner` on remote-cache hits in a large monorepo.

Tradeoffs and open questions:

- More specialized around tests and their executable producer.
- Introduces an alternate test identity that Bazel must compute consistently.
- Eligibility must be restricted to deterministic, cacheable producers.
- The current prototype does not support merged Skyframe analysis/execution.

## Questions for maintainers

We would appreciate guidance on the intended abstraction for this problem.

We are happy to revise either prototype or explore another approach. The main goal of this issue is to agree on the model before investing further in a particular implementation.

Contributor guide

Open the contributing guide

Research direction

Start by reading the two referenced prototypes, PRs 30635 and 30636, and compare their behavior with --remote_download_minimal. Exercise downstream cache hits and misses around GoLink and TestRunner, including the merged Skyframe limitation. Done means maintainers agree on an abstraction and its invalidation and fallback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, java
Domain
build-system, performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.