Don't download repositories when both remote execution and remote downloader are enabled
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
When both remote execution and remote downloader are enabled what essentially happens is repositories are downloaded locally from remote cache just to be subsequently uploaded back to the remote cache for remote execution. Since with remote downloader enabled repositories are already inside the remote cache and have digest ids they can be pulled directly from the remote cache by the remote executor. No heavy traffic needs to flow between the bazel client and the build cluster, bazel can get the digest id of the repository file and forward it directly to the remote executor without downloading or uploading any files. Currently those repository files travel the network (possibly twice) for no reason at all.
I have a repository with 12G `external/` directory and downloading them on every CI build is a bit of an issue so I have to cache them in the docker image. Which is fine, but then the image is 12G heavier and pulling it every time a new node is spawned takes a long time.
This is still happening on Bazel 3.5.0.
Contributor guide
Assessment
This issue has not been assessed yet.