bazelbuild / bazelbuild/bazel

Support internal dependencies that are resolved on-disk via another tool

Open
#16,380 49 comments 3 reactions 1 assignee Claimed by @michaeledgar View on GitHub
P4 team-Core type: feature request untriaged
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

### Description of the feature request:

Add "filesystem adapter" API(s) so that when a source file isn't found on-disk during the [loading phase][bazel-loading-phase], Bazel can fall back on a user-specified function to vivify the file (if it exists) rather than throwing an error.

This request comes from a desire to have Bazel work in a Git [sparse-checkout][git-sparse-checkout] environment _without_ pre-specifying patterns (see below), but the feature probably shouldn't be scoped to just Git; for example, Mercurial has a similar [sparse checkout
feature][hg-sparse-checkout]. I'd be happy to see a feature like this be even more general purpose if there are other cases it could support!

[bazel-loading-phase]: https://bazel.build/reference/glossary#loading-phase
[git-sparse-checkout]: https://git-scm.com/docs/git-sparse-checkout
[hg-sparse-checkout]: https://firefox-source-docs.mozilla.org/build/buildsystem/sparse.html

### What underlying problem are you trying to solve with this feature?

[Internal dependencies][bazel-internal-deps] in Bazel are implicitly required to be on-disk when performing Bazel operations (`query`, `build`, etc.) that involve loading the associated source files. However, in a Git sparse-checkout environment, source files are not present in the worktree (i.e., where Bazel is looking for them) unless vivified with `git sparse-checkout add`. Even when not on-disk, though, these source files are more consistent with internal dependencies (fixed to the same "version" as the rest of the repo, not a separate pre-built package, are checked out to source directories) than external ones.

Right now, the only way to work in a sparse-checkout with Bazel is to specify the files that a particular build will need _before_ running `bazel build`. A user has to maintain two independent sources of truth for which files are needed in a build: the Bazel dependency graph, and the sparse-checkout patterns. If the not-on-disk sparse-checkout files can reasonably be interpreted as "internal dependencies" without running contrary to Bazel's underlying principles, it would be extremely helpful to users if they could resolve the files "just-in-time" while Bazel builds its internal dependency graph.

[bazel-internal-deps]: https://bazel.build/basics/dependencies#internal_dependencies

### Which operating system are you running Bazel on?

_No response_

### What is the output of `bazel info release`?

_No response_

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

_No response_

### Have you found anything relevant by searching the web?

Within Bazel itself, it looks like the only other reference to `sparse-checkout` is a feature request to include a static pattern list in the specification of a `git_repository()`. That request doesn't quite match this one, since 1) that deals with an external repository, rather than the local one, and 2) it still requires pre-determined patterns, rather than resolving files just-in-time.

Outside of Bazel, third-party Bazel/sparse-checkout interoperability tools are pretty common, especially among large monorepo projects ([1][canva-blog], [2][twitter-focus-talk]). However, these tools still rely on a fully on-disk copy of a Git repository to generate the Bazel dependency graph, severely inhibiting the potential performance gains of `git sparse-checkout`. And, because these integrations use a tool separate from either `git` or `bazel`,
they are potentially more difficult to adopt than an integration contained in a Bazel rule library.

[canva-blog]: https://canvatechblog.com/we-put-half-a-million-files-in-one-git-repository-heres-what-we-learned-ec734a764181
[twitter-focus-talk]: https://docs.google.com/presentation/d/12RVWPIms-rFKfteqYa5bpSWElIiJ1oCAYobLb5DihQo/edit?usp=sharing

### Any other information, logs, or outputs that you want to share?

I know this is a pretty large feature request (with lots of details to consider: UX, performance, thread safety, etc.), so if this request doesn't seem completely infeasible, I'm happy to write & submit a more detailed [design doc][bazel-design-doc]!

[bazel-design-doc]: https://bazel.build/contribute/design-documents

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.