bazelbuild / bazelbuild/bazel

Allow git_override to specify pathspec for --recurse_submodules

Open
#21,186 2 comments 0 reactions 0 assignees View on GitHub
help wanted P2 team-ExternalDeps type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

**The feature request**

We request to add an attribute to `git_override`:

```python
git_override(
module_name = "kleaf",
remote = "https://android.googlesource.com/kernel/superproject",
commit = "1c03b74f03bb7a45856b8e131a857c238af12033",
init_submodules = True,
recurse_submodules_pathspec = ":(attr:ddk)"
)
```
... or any solution that can allow us to pull only submodules with the given attribute to achieve the behavior of `git clone --recurse-submodules=':(attr:ddk)' ...`

### Which category does this issue belong to?

External Dependency

### What underlying problem are you trying to solve with this feature?

**Background**
We (the [Kleaf](https://android.googlesource.com/kernel/build/+/refs/heads/main/kleaf/README.md) team) are working on a feature that needs to pull a subtree of a source tree as a module that builds a build number on ci.android.com. The way we design it is the following:

- From ci.android.com, for a given build number, infer the superproject SHA of the [kernel superproject](https://android.googlesource.com/kernel/superproject)
- Add the following to MODULE.bazel:

```python
git_override(
module_name = "kleaf",
remote = "https://android.googlesource.com/kernel/superproject",
commit = "1c03b74f03bb7a45856b8e131a857c238af12033",
init_submodules = True,
)
```

However, this will pull the whole superproject. We would like to be able to pull only projects with certain .gitattributes ([example](https://android.googlesource.com/kernel/superproject/+/refs/heads/common-android-mainline/.gitattributes). These git attributes were originally from the `groups` of a repo manifest ([example](https://android.googlesource.com/kernel/manifest/+/refs/heads/common-android-mainline/default.xml#8))

So, to pull groups=ddk projects, with `repo` we would do
```shell
repo init --groups=ddk ...
```
With Git submodules we would do
```shell
git clone --recurse-submodules=':(attr:ddk)' ...
```

However, there is no way to specify `--recurse-submodules=` in `git_override`.

**Alternatives**

One may suggest us to use `use_repo_rule` and implement our own repo_rule instead. However, I need a full module so that bazel will read its `MODULE.bazel` and resolve the dependencies, not just a single repository.

We also come up with some hacky alternative solutions, example:

```python
git_override(
module_name = "kleaf",
remote = "https://android.googlesource.com/kernel/superproject",
commit = "1c03b74f03bb7a45856b8e131a857c238af12033",
patch_cmds = [
"tools/filter_submodules.sh -g ddk",
],
)
```

where the `tools/filter_submodules.sh` contains:

```sh
GROUP=ddk #

git rm -r $(git check-attr $GROUP -- $(git submodule status | cut -f2 -d' ') | \
grep -v "$GROUP: set" | cut -f1 -d':')

git commit -am'Remove unrelated projects'

git submodule update --init --recursive
```

### Which operating system are you running Bazel on?

Linux

### What is the output of `bazel info release`?

release 7.0.2

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

_No response_

### Have you found anything relevant by searching the web?

Git pathspecs: https://css-tricks.com/git-pathspecs-and-how-to-use-them/#aa-attr
Git submodules: https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---recurse-submodulesltpathspecgt

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the git_override entry point and its existing init_submodules handling; compare the submodule invocation with Git's documented --recurse-submodules= behavior. Done means git_override can select submodules by pathspec while preserving module loading, with coverage for the requested usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
git
Domain
build-system
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.