Allow starlark rules to filter File sets using `glob`.
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 75
Description
### Description of the feature request:
Give starlark rule implementations the ability to filter a `depset` or `list` of `File` objects using [glob](https://bazel.build/reference/be/functions#glob). This should match based on the `short_path` of the file, and expose both `include` and `exclude` pattern sets.
### What underlying problem are you trying to solve with this feature?
There are various cases where it's useful to be able to filter a set of files, thus the existence of for example skylib's [`select_file`](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/select_file.bzl) rule. In our build, we want to be able to exclude certain generated files from a final tarball output, and especially with remote execution it's more efficient to do the starlark in the rule implementation rather than letting them get to the remote executor before being ignored (it's also better for caching).
In an ideal world, this would be handled by the upstream rules providing various output groups, so the consuming rule could build the set of files additively rather than subtractive, however from a practical perspective this isn't always possible, as the upstream rules are often not under a particular rule author's control and/or propagating the divisions between output groups through intermediate layers of rules depending on the upstream rules might be very complex.
Currently, the way this sort of thing is implemented (in `select_file`, as well as in our internal rules) is to take the `depset`, call `to_list` on it, and then filter the result. There's at least two problems with this approach:
1. `depset.to_list()` is inefficient, particularly when (as is the case here) we don't intend to keep the entire result. Having the filtering done outside of starlark would be at least potentially more efficient. For example in the case of `select_file` the rule is expecting to only match a single file from the `depset`, which might potentially contain thousands of input files.
2. Starlark doesn't have any kind of built-in functionality for testing `glob` or `regexp` matches in starlark rules. This means our internal rules needed to "fake it" by using a sequence of `split`, `startswith`, and `endswith` expressions. This works (mostly; there are some edge cases which are difficult to handle), but is computationally expensive, especially if one wishes to test vs several patterns.
One could potentially solve problem 2 without solving problem 1, by providing either a `regexp` package or a function to test a string against a `glob` pattern (similar to go's [`filepath.Match`](https://pkg.go.dev/path/filepath#Match)), which would significantly simplify `to_list()`-based implementations, and probably be more performant. This might be a good idea in any case, but I strongly suspect that the primary use cases for at least the `glob` matching functionality would be better served by allowing it directly filter a `depset`.
### Which operating system are you running Bazel on?
linux
### What is the output of `bazel info release`?
release 5.2.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?
_No response_
### Have you found anything relevant by searching the web?
Implementation of matching for `select_file`: https://github.com/bazelbuild/bazel-skylib/blob/a501641daebdce8601ee6bbc6b5d177c688d3517/rules/select_file.bzl#L27-L30
https://github.com/google/starlark-go/issues/241 proposes exposing a `regexp` module in starlark. [qri](https://qri.io/docs/reference/starlark-packages/re)'s starlark stdlib actually implements that.
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start by reviewing the Starlark glob documentation and skylib's select_file.bzl implementation, especially its current depset.to_list filtering. Trace Bazel's depset and File entry points to determine where include and exclude matching could belong. Done means Starlark rules can filter File collections by short_path with both pattern sets, with behavior and edge cases covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100