bazelbuild / bazelbuild/bazel

Remote repo contents cache fails to recover from missing CAS blobs during repository materialization

Open
#30,218 1 comment 0 reactions 1 assignee Claimed by @fmeum View on GitHub
P2 team-ExternalDeps type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

The remote repo contents cache does not recover when a cached repository's `Tree` references a leaf blob that is no longer in the content-addressable store. If another repository rule triggers full materialization through `rctx.path()` or `rctx.read()`, the command fails with `Failed to materialize remote repo` and `Missing digest`.

A cache hit downloads the repository marker and `Tree`, then injects the `Tree` into `RemoteExternalOverlayFileSystem`. Bazel does not verify that every leaf digest in the `Tree` is still available. During later full materialization, `RemoteExternalOverlayFileSystem.doMaterialize()` prefetches the leaves and propagates the missing-digest error. Bazel's default whole-command remote cache eviction retries do not invalidate and refetch the repository in this case.

Expected behavior: Bazel discards the unusable repository-cache entry and runs the repository rule again.

Actual behavior: the command fails after the default retries, and the repository rule does not run again.

### Which category does this issue belong to?

External Dependency, Remote Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

A self-contained integration reproducer is available in [`RemoteRepoContentsCacheTest.testReproMissingLeafDuringFullRepoMaterializationIsFatal`](https://github.com/michaelm-openai/bazel/blob/bd6249f9ed49cadae412f5d57ec77de45ab6cc3f/src/test/py/bazel/bzlmod/remote_repo_contents_cache_test.py#L1737-L1807).

The test:

1. Creates a reproducible repository containing `data.txt`.
2. Populates the remote repo contents cache.
3. Runs `bazel clean --expunge` and restores only the repository metadata into the overlay file system.
4. Deletes the CAS blob for `data.txt` while retaining the repository action result and `Tree`.
5. Evaluates another repository rule that reads `data.txt` with `rctx.path()` and `rctx.read()`.
6. Verifies that the command fails with `Failed to materialize remote repo` and `Missing digest`, without rerunning the original repository rule.

With commit `bd6249f9ed49cadae412f5d57ec77de45ab6cc3f` checked out, run:

```shell
bazel test //src/test/py/bazel:remote_repo_contents_cache_test \
--test_filter=RemoteRepoContentsCacheTest.testReproMissingLeafDuringFullRepoMaterializationIsFatal \
--test_sharding_strategy=disabled \
--nocache_test_results \
--test_output=errors
```

The integration test treats the inner build failure as the observed bug, so the test passes when the bug is reproduced.

### Which operating system are you running Bazel on?

Ubuntu 24.04.4 LTS, x86_64.

### What is the output of `bazel info release`?

`development version`

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

Built from `bazelbuild/bazel` at `632d28211321b01d897829c078897158db313f51`, the `9.2.0rc2` tag, using Bazel 9.1.1. The integration test runs the self-built Bazel binary.

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text
https://github.com/bazelbuild/bazel.git
bd6249f9ed49cadae412f5d57ec77de45ab6cc3f
```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

This is not known to be a regression. The reproducer was tested against 9.2.0rc2. No earlier version was tested.

### Have you found anything relevant by searching the web?

The existing [`testLostRemoteFile_build`](https://github.com/bazelbuild/bazel/blob/9.2.0rc2/src/test/py/bazel/bzlmod/remote_repo_contents_cache_test.py#L1574-L1659) covers a missing repository file needed during package loading. That path marks the repository as having lost files, retries the build, and refetches the repository. It does not cover a missing leaf discovered by full repository materialization.

A search for remote repo contents cache issues containing `Missing digest` or repository materialization did not find an existing report for this case.

### Any other information, logs, or outputs that you want to share?

The cache-hit path downloads only the marker and `Tree` before injecting the repository: [`RemoteRepoContentsCacheImpl.java`](https://github.com/bazelbuild/bazel/blob/9.2.0rc2/src/main/java/com/google/devtools/build/lib/remote/RemoteRepoContentsCacheImpl.java#L256-L297).

Full materialization fetches every leaf through [`RemoteExternalOverlayFileSystem.doMaterialize()`](https://github.com/bazelbuild/bazel/blob/9.2.0rc2/src/main/java/com/google/devtools/build/lib/remote/RemoteExternalOverlayFileSystem.java#L301-L351). The separate on-demand file-read path marks the repository for invalidation when a leaf has been evicted at [`RemoteExternalOverlayFileSystem.java`](https://github.com/bazelbuild/bazel/blob/9.2.0rc2/src/main/java/com/google/devtools/build/lib/remote/RemoteExternalOverlayFileSystem.java#L747-L769).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.