bazelbuild / bazelbuild/bazel

Allow `git_repository()` to clone from a local look-aside cache.

Open
#17,600 6 comments 0 reactions 0 assignees View on GitHub
help wanted P2 team-ExternalDeps type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

What I'd like is some principled application of these concepts by `git_repository()`:

https://randyfay.com/content/reference-cache-repositories-speed-clones-git-clone-reference
https://randyfay.com/content/git-clone-reference-considered-harmful
https://www.alchemists.io/articles/git_metadata_cloning

**tl;dr;** allow using a slow git remote as an authoritative source of truth while diverting as much of the bandwidth to more local and faster git sources. (For all I know `--disk_cache` et al. do that, but observational evidence suggests that's not the case.) Both local disk and on-premise git servers would be desirable caching options.

Ideally this could show up as a `string_list_flag()` that can be set via `~/.bazelrc`, `--bazelrc` or the like with valid git remote URLs. As much as practical, `git_repository()` would use them for data and only fall back on `git_repository().remote` when it needs to "page in" data to the caches that's missing (or non-authoritative). Importantly, changing or removing that URL list or making them inaccessible should not alter build results.

---
The longer version of the idea (i.e. my somehwat ignorant musings about how I'd go about trying to implant this) is some way to avoid re-downloading git repos across the internet every time a "clean" build is kicked off. Given git's implementation, this could in theory be done by using a git repo somewhere "local" that includes all the remotes bazel has been asked to clone from and then pulling the majority of data from that.

Someone who knows a lot more about git internals than I do could probably figure out how to use one remote to resolve the commit to clone and then another (priority ordered list of) remote(s) to actually fetch the content. A possibility (that would require the cache actually be local) would be to first resolve commits and fetch into a cache directory (adding new remotes as needed) before doing the normal fetch with that direct as a substituted remote. Another possibility would be some kind of shallow, blobless and/or treeless clone from the real remote, followed by filling in the trees/blobs/etc. from the local/cache remote (where possible) and only pulling blobs/trees from the real remote as a last resort (and pushing them to the local/cache for next time).

**Side note:** Much of the work needed to make this happen would overlap with allowing `git_repository()` to use a *list* of remotes for fault tolerance and load spreading. As long as `.commit` is used this wouldn't even have issues with consistency.

### What underlying problem are you trying to solve with this feature?

CI builds (and any other processes that start clean/clean'ish builds) can spend a significant amount of time just fetching sources.

### Which operating system are you running Bazel on?

any

### What is the output of `bazel info release`?

any

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

n/a

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

```text
n/a
```

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading the git_repository rule and the existing --disk_cache behavior, then review how Bazel accepts configuration from .bazelrc files. Define the scope for priority-ordered local or on-premise Git remotes, including fallback to the authoritative remote and unchanged build results when caches change or are unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
git
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.