`--experimental_remote_repo_contents_cache` unnecessarily materializes entire repos to local disk
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
We have a large monorepo with a CI step that essentially does `bazel build //... --config=remote --remote_download_minimal` on a clean GHA worker (BYOB).
We are trying to enable `--experimental_remote_repo_contents_cache` to avoid wasting time in downloading and unpacking giant tarballs locally (only our llvm toolchain takes 4-5 minutes to set up on every CI run...).
However, we are seeing a performance _regression_ enabling this during the analysis phase (8->13 min). The problem seems to be the need of materializing _45 repos_ from the RBE CAS over gRPC file by file, which is slower than downloading and extracting tarballs.
I believe the issue comes from Bzlmod module extensions needing to inspect metadata files in other modules:
* Gazelle (go_deps): Reads go.mod files from contributing Go modules (eg @rules_go//:go.mod).
* rules_jvm_external (maven): Reads POM or metadata files from contributing modules (@protobuf, @grpc).
* rules_rust (crate_universe): Reads Cargo.toml / Cargo.lock files from external crates.
Instead of reading these individual files, we do a _full materialization_ of the entire repos onto the local disk. In the debug output we see dozens of repositories log `DEBUG: Materializing remote repo @@+`
I believe the issue is that `readFile` is eventually hitting `remoteFS.ensureMaterialized` [here](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkBaseExternalContext.java#L2405)
This seems like an expensive behavior of a `getPathFromLabel` function and wasteful given Bazel [can already](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/remote/RemoteExternalOverlayFileSystem.java#L889-L898) read individual blobs from the remoteFS into memory.
### Which category does this issue belong to?
Remote Execution
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
In a large repo using Bzlmod (with Gazelle, rules_jvm_external, or rules_rust), enable: `startup --experimental_remote_repo_contents_cache` and compare analysis behavior and debug output on a clean state (`bazel clean --expunge`)
### Which operating system are you running Bazel on?
Linux
### What is the output of `bazel info release`?
release 8.8.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
private repo
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
_No response_
### Have you found anything relevant by searching the web?
No
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.