bazelbuild / bazelbuild/bazel

Allow starlark rules to filter File sets using `glob`.

Open
#15,659 1 comment 2 reactions 0 assignees View on GitHub
P2 team-Rules-API type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

### Description of the feature request:

Give starlark rule implementations the ability to filter a `depset` or `list` of `File` objects using [glob](https://bazel.build/reference/be/functions#glob). This should match based on the `short_path` of the file, and expose both `include` and `exclude` pattern sets.

### What underlying problem are you trying to solve with this feature?

There are various cases where it's useful to be able to filter a set of files, thus the existence of for example skylib's [`select_file`](https://github.com/bazelbuild/bazel-skylib/blob/main/rules/select_file.bzl) rule. In our build, we want to be able to exclude certain generated files from a final tarball output, and especially with remote execution it's more efficient to do the starlark in the rule implementation rather than letting them get to the remote executor before being ignored (it's also better for caching).

In an ideal world, this would be handled by the upstream rules providing various output groups, so the consuming rule could build the set of files additively rather than subtractive, however from a practical perspective this isn't always possible, as the upstream rules are often not under a particular rule author's control and/or propagating the divisions between output groups through intermediate layers of rules depending on the upstream rules might be very complex.

Currently, the way this sort of thing is implemented (in `select_file`, as well as in our internal rules) is to take the `depset`, call `to_list` on it, and then filter the result. There's at least two problems with this approach:

1. `depset.to_list()` is inefficient, particularly when (as is the case here) we don't intend to keep the entire result. Having the filtering done outside of starlark would be at least potentially more efficient. For example in the case of `select_file` the rule is expecting to only match a single file from the `depset`, which might potentially contain thousands of input files.
2. Starlark doesn't have any kind of built-in functionality for testing `glob` or `regexp` matches in starlark rules. This means our internal rules needed to "fake it" by using a sequence of `split`, `startswith`, and `endswith` expressions. This works (mostly; there are some edge cases which are difficult to handle), but is computationally expensive, especially if one wishes to test vs several patterns.

One could potentially solve problem 2 without solving problem 1, by providing either a `regexp` package or a function to test a string against a `glob` pattern (similar to go's [`filepath.Match`](https://pkg.go.dev/path/filepath#Match)), which would significantly simplify `to_list()`-based implementations, and probably be more performant. This might be a good idea in any case, but I strongly suspect that the primary use cases for at least the `glob` matching functionality would be better served by allowing it directly filter a `depset`.

### Which operating system are you running Bazel on?

linux

### What is the output of `bazel info release`?

release 5.2.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

_No response_

### Have you found anything relevant by searching the web?

Implementation of matching for `select_file`: https://github.com/bazelbuild/bazel-skylib/blob/a501641daebdce8601ee6bbc6b5d177c688d3517/rules/select_file.bzl#L27-L30

https://github.com/google/starlark-go/issues/241 proposes exposing a `regexp` module in starlark. [qri](https://qri.io/docs/reference/starlark-packages/re)'s starlark stdlib actually implements that.

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Starlark glob documentation and skylib's select_file.bzl implementation, especially its current depset.to_list filtering. Trace Bazel's depset and File entry points to determine where include and exclude matching could belong. Done means Starlark rules can filter File collections by short_path with both pattern sets, with behavior and edge cases covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.