bazelbuild / bazelbuild/bazel

Expose TransitiveTestsProvider to Starlark

Open
#14,993 6 comments 1 reaction 0 assignees View on GitHub
P3 stale team-Rules-API type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the problem / feature request:

> `native.test_suite` rules only accept `tests` dependencies that are other `test_suite`s or actual `*_test` targets. It would be very helpful to have `test_suite` expose a provider for `tests` so that custom rules can be used as dependencies.

### Feature requests: what underlying problem are you trying to solve with this feature?

> Bazel test suites can be hierarchical: they can either depend on other test suites or test targets themselves. When using tags as filtration mechanisms, “filtering by specified `tags` is only done for tests listed directly in this attribute. If this attribute contains `test_suites`s, the test inside those will not be filtered by this `test_suite` (they are considered to be filtered already)”. This behavior is not ideal in all cases. Consider the following:

```
test_suite(
name=‘precheckin_tests’,
tests=[
‘//a/b/c:d’, # size=small
‘//e/f:g’, # size=medium
‘//w/x/y:z’, # size=large
],
)

# The following doesn’t work, and will run all tests in precheckin_tests, regardless of size
test_suite(
name=‘fast_precheckin_tests’,
tests=[‘:precheckin_tests’],
tags=[‘small’],
)
```

> In order to get around this issue, for each `test_suite` target, I wanted to automatically expand its `test_suite` dependencies into leaf `*_test` targets, so that filtration as above can work. To do this, I created a Bazel rule & aspect that recursively search through listed `tests` in `test_suite` and gathered all the leaf labels. This works, but *there is no way to use the new rule as a dependency of `test_suite`, because `test_suite` doesn’t expose any providers*.

```
TestInfo = provider(
fields = {
‘leaf_tests’: ‘underlying test targets for hierarchical native.test_suites`,
}
)

def _test_suite_leaf_tests_aspect_impl(target, ctx):
if hasattr(ctx.rule.attr, ‘tests’):
# Recursive test suite, gather labels of dependencies
leaf_tests = depset(
direct=[],
transitive=[t[TestInfo].leaf_tests for t in ctx.rule.attr.tests],
)
return [TestInfo(leaf_tests=leaf_tests)]
else:
# Test target, just gather its label
leaf_tests = depset(direct=[ctx.label])
return [TestInfo(leaf_tests=leaf_tests)]

test_suite_leaf_tests_aspect = aspect(
implementation=_test_suite_leaf_tests_aspect_impl,
attr_aspects=[‘tests’],
)

def _test_suite_leaf_tests_rule_impl(ctx):
leaf_tests = depset(transitive=[t[TestInfo].leaf_tests for t in ctx.attr.tests])
# Outputs correct list
print(leaf_tests.to_list())
# This cannot be used by native.test_suite :(
return [TestInfo(leaf_tests=leaf_tests)]

test_suite_leaf_tests = rule(
implementation=_test_suite_leaf_tests_rule_impl,
attrs = {
‘tests’: attr.label_list(aspects=[test_suite_leaf_tests_aspect])
}
)
```

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

> N/A

### What operating system are you running Bazel on?

> RHEL7

### What's the output of `bazel info release`?

> release 0.24.0- (@non-git)

### If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.

> Company-specific, but behavior matches Bazel documentation

### What's the output of `git remote get-url origin ; git rev-parse master ; git rev-parse HEAD` ?

> Company-specific, but behavior matches Bazel documentation

### Have you found anything relevant by searching the web?

> Documented Bazel providers (https://docs.bazel.build/versions/main/skylark/lib/skylark-provider.html) and reading source code in `bazel/src/main/java/com/google/de tools/build/lib/rules/test/*` indicates that the internal `TransitiveTestProvider` is not exposed to StarLark

### Any other information, logs, or outputs that you want to share?

> N/A

Contributor guide

Open the contributing guide

Research direction

Start with native.test_suite and the test-rule sources under bazel/src/main/java/com/google/de tools/build/lib/rules/test/*, then compare the internal TransitiveTestProvider with the documented Starlark providers. Reproduce the hierarchical test_suite example and verify that a custom rule can consume the exposed provider so tag filtering reaches leaf tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.