Remote repo contents cache fails to recover from missing CAS blobs during repository materialization
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
The remote repo contents cache does not recover when a cached repository's `Tree` references a leaf blob that is no longer in the content-addressable store. If another repository rule triggers full materialization through `rctx.path()` or `rctx.read()`, the command fails with `Failed to materialize remote repo` and `Missing digest`.
A cache hit downloads the repository marker and `Tree`, then injects the `Tree` into `RemoteExternalOverlayFileSystem`. Bazel does not verify that every leaf digest in the `Tree` is still available. During later full materialization, `RemoteExternalOverlayFileSystem.doMaterialize()` prefetches the leaves and propagates the missing-digest error. Bazel's default whole-command remote cache eviction retries do not invalidate and refetch the repository in this case.
Expected behavior: Bazel discards the unusable repository-cache entry and runs the repository rule again.
Actual behavior: the command fails after the default retries, and the repository rule does not run again.
### Which category does this issue belong to?
External Dependency, Remote Execution
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
A self-contained integration reproducer is available in [`RemoteRepoContentsCacheTest.testReproMissingLeafDuringFullRepoMaterializationIsFatal`](https://github.com/michaelm-openai/bazel/blob/bd6249f9ed49cadae412f5d57ec77de45ab6cc3f/src/test/py/bazel/bzlmod/remote_repo_contents_cache_test.py#L1737-L1807).
The test:
1. Creates a reproducible repository containing `data.txt`.
2. Populates the remote repo contents cache.
3. Runs `bazel clean --expunge` and restores only the repository metadata into the overlay file system.
4. Deletes the CAS blob for `data.txt` while retaining the repository action result and `Tree`.
5. Evaluates another repository rule that reads `data.txt` with `rctx.path()` and `rctx.read()`.
6. Verifies that the command fails with `Failed to materialize remote repo` and `Missing digest`, without rerunning the original repository rule.
With commit `bd6249f9ed49cadae412f5d57ec77de45ab6cc3f` checked out, run:
```shell
bazel test //src/test/py/bazel:remote_repo_contents_cache_test \
--test_filter=RemoteRepoContentsCacheTest.testReproMissingLeafDuringFullRepoMaterializationIsFatal \
--test_sharding_strategy=disabled \
--nocache_test_results \
--test_output=errors
```
The integration test treats the inner build failure as the observed bug, so the test passes when the bug is reproduced.
### Which operating system are you running Bazel on?
Ubuntu 24.04.4 LTS, x86_64.
### What is the output of `bazel info release`?
`development version`
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
Built from `bazelbuild/bazel` at `632d28211321b01d897829c078897158db313f51`, the `9.2.0rc2` tag, using Bazel 9.1.1. The integration test runs the self-built Bazel binary.
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
https://github.com/bazelbuild/bazel.git
bd6249f9ed49cadae412f5d57ec77de45ab6cc3f
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
This is not known to be a regression. The reproducer was tested against 9.2.0rc2. No earlier version was tested.
### Have you found anything relevant by searching the web?
The existing [`testLostRemoteFile_build`](https://github.com/bazelbuild/bazel/blob/9.2.0rc2/src/test/py/bazel/bzlmod/remote_repo_contents_cache_test.py#L1574-L1659) covers a missing repository file needed during package loading. That path marks the repository as having lost files, retries the build, and refetches the repository. It does not cover a missing leaf discovered by full repository materialization.
A search for remote repo contents cache issues containing `Missing digest` or repository materialization did not find an existing report for this case.
### Any other information, logs, or outputs that you want to share?
The cache-hit path downloads only the marker and `Tree` before injecting the repository: [`RemoteRepoContentsCacheImpl.java`](https://github.com/bazelbuild/bazel/blob/9.2.0rc2/src/main/java/com/google/devtools/build/lib/remote/RemoteRepoContentsCacheImpl.java#L256-L297).
Full materialization fetches every leaf through [`RemoteExternalOverlayFileSystem.doMaterialize()`](https://github.com/bazelbuild/bazel/blob/9.2.0rc2/src/main/java/com/google/devtools/build/lib/remote/RemoteExternalOverlayFileSystem.java#L301-L351). The separate on-demand file-read path marks the repository for invalidation when a leaf has been evicted at [`RemoteExternalOverlayFileSystem.java`](https://github.com/bazelbuild/bazel/blob/9.2.0rc2/src/main/java/com/google/devtools/build/lib/remote/RemoteExternalOverlayFileSystem.java#L747-L769).
Contributor guide
Assessment
This issue has not been assessed yet.