Allow `git_repository()` to clone from a local look-aside cache.
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the feature request:
What I'd like is some principled application of these concepts by `git_repository()`:
https://randyfay.com/content/reference-cache-repositories-speed-clones-git-clone-reference
https://randyfay.com/content/git-clone-reference-considered-harmful
https://www.alchemists.io/articles/git_metadata_cloning
**tl;dr;** allow using a slow git remote as an authoritative source of truth while diverting as much of the bandwidth to more local and faster git sources. (For all I know `--disk_cache` et al. do that, but observational evidence suggests that's not the case.) Both local disk and on-premise git servers would be desirable caching options.
Ideally this could show up as a `string_list_flag()` that can be set via `~/.bazelrc`, `--bazelrc` or the like with valid git remote URLs. As much as practical, `git_repository()` would use them for data and only fall back on `git_repository().remote` when it needs to "page in" data to the caches that's missing (or non-authoritative). Importantly, changing or removing that URL list or making them inaccessible should not alter build results.
---
The longer version of the idea (i.e. my somehwat ignorant musings about how I'd go about trying to implant this) is some way to avoid re-downloading git repos across the internet every time a "clean" build is kicked off. Given git's implementation, this could in theory be done by using a git repo somewhere "local" that includes all the remotes bazel has been asked to clone from and then pulling the majority of data from that.
Someone who knows a lot more about git internals than I do could probably figure out how to use one remote to resolve the commit to clone and then another (priority ordered list of) remote(s) to actually fetch the content. A possibility (that would require the cache actually be local) would be to first resolve commits and fetch into a cache directory (adding new remotes as needed) before doing the normal fetch with that direct as a substituted remote. Another possibility would be some kind of shallow, blobless and/or treeless clone from the real remote, followed by filling in the trees/blobs/etc. from the local/cache remote (where possible) and only pulling blobs/trees from the real remote as a last resort (and pushing them to the local/cache for next time).
**Side note:** Much of the work needed to make this happen would overlap with allowing `git_repository()` to use a *list* of remotes for fault tolerance and load spreading. As long as `.commit` is used this wouldn't even have issues with consistency.
### What underlying problem are you trying to solve with this feature?
CI builds (and any other processes that start clean/clean'ish builds) can spend a significant amount of time just fetching sources.
### Which operating system are you running Bazel on?
any
### What is the output of `bazel info release`?
any
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
n/a
### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?
```text
n/a
```
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start by reading the git_repository rule and the existing --disk_cache behavior, then review how Bazel accepts configuration from .bazelrc files. Define the scope for priority-ordered local or on-premise Git remotes, including fallback to the authoritative remote and unchanged build results when caches change or are unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100