Support internal dependencies that are resolved on-disk via another tool
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 75
Description
### Description of the feature request:
Add "filesystem adapter" API(s) so that when a source file isn't found on-disk during the [loading phase][bazel-loading-phase], Bazel can fall back on a user-specified function to vivify the file (if it exists) rather than throwing an error.
This request comes from a desire to have Bazel work in a Git [sparse-checkout][git-sparse-checkout] environment _without_ pre-specifying patterns (see below), but the feature probably shouldn't be scoped to just Git; for example, Mercurial has a similar [sparse checkout
feature][hg-sparse-checkout]. I'd be happy to see a feature like this be even more general purpose if there are other cases it could support!
[bazel-loading-phase]: https://bazel.build/reference/glossary#loading-phase
[git-sparse-checkout]: https://git-scm.com/docs/git-sparse-checkout
[hg-sparse-checkout]: https://firefox-source-docs.mozilla.org/build/buildsystem/sparse.html
### What underlying problem are you trying to solve with this feature?
[Internal dependencies][bazel-internal-deps] in Bazel are implicitly required to be on-disk when performing Bazel operations (`query`, `build`, etc.) that involve loading the associated source files. However, in a Git sparse-checkout environment, source files are not present in the worktree (i.e., where Bazel is looking for them) unless vivified with `git sparse-checkout add`. Even when not on-disk, though, these source files are more consistent with internal dependencies (fixed to the same "version" as the rest of the repo, not a separate pre-built package, are checked out to source directories) than external ones.
Right now, the only way to work in a sparse-checkout with Bazel is to specify the files that a particular build will need _before_ running `bazel build`. A user has to maintain two independent sources of truth for which files are needed in a build: the Bazel dependency graph, and the sparse-checkout patterns. If the not-on-disk sparse-checkout files can reasonably be interpreted as "internal dependencies" without running contrary to Bazel's underlying principles, it would be extremely helpful to users if they could resolve the files "just-in-time" while Bazel builds its internal dependency graph.
[bazel-internal-deps]: https://bazel.build/basics/dependencies#internal_dependencies
### Which operating system are you running Bazel on?
_No response_
### What is the output of `bazel info release`?
_No response_
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?
_No response_
### Have you found anything relevant by searching the web?
Within Bazel itself, it looks like the only other reference to `sparse-checkout` is a feature request to include a static pattern list in the specification of a `git_repository()`. That request doesn't quite match this one, since 1) that deals with an external repository, rather than the local one, and 2) it still requires pre-determined patterns, rather than resolving files just-in-time.
Outside of Bazel, third-party Bazel/sparse-checkout interoperability tools are pretty common, especially among large monorepo projects ([1][canva-blog], [2][twitter-focus-talk]). However, these tools still rely on a fully on-disk copy of a Git repository to generate the Bazel dependency graph, severely inhibiting the potential performance gains of `git sparse-checkout`. And, because these integrations use a tool separate from either `git` or `bazel`,
they are potentially more difficult to adopt than an integration contained in a Bazel rule library.
[canva-blog]: https://canvatechblog.com/we-put-half-a-million-files-in-one-git-repository-heres-what-we-learned-ec734a764181
[twitter-focus-talk]: https://docs.google.com/presentation/d/12RVWPIms-rFKfteqYa5bpSWElIiJ1oCAYobLb5DihQo/edit?usp=sharing
### Any other information, logs, or outputs that you want to share?
I know this is a pretty large feature request (with lots of details to consider: UX, performance, thread safety, etc.), so if this request doesn't seem completely infeasible, I'm happy to write & submit a more detailed [design doc][bazel-design-doc]!
[bazel-design-doc]: https://bazel.build/contribute/design-documents
Contributor guide
Assessment
This issue has not been assessed yet.