Top-level aspects cannot analyze an unexpanded test_suite with incompatible members
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
`--noexpand_test_suites` is suggested for top-level aspects because it allows an aspect to analyze the `test_suite` target itself, but it doesn't work with incompatible members.
Bazel marks the unexpanded `test_suite` indirectly incompatible before invoking the aspect, so it can't see the compatible members, or produce its output group.
With `--expand_test_suites` it correctly skips the incompatible test, but the aspect is then applied separately to each expanded test and there is no suite-level aspect application.
I would have expected that top-level aspects could analyze an unexpanded `test_suite` and skip incompatible members, consistent with normal test-suite expansion. Maybe instead aspects could have an API for applying to incompatible targets and inspecting incompatibility, like the `apply_to_incompatible_targets` idea in ?
Currently, the indirect-incompatibility check makes the entire unexpanded suite incompatible. Since Bazel doesn't apply aspects to incompatible targets, the aspect can't filter the incompatible member itself.
### Which category does this issue belong to?
Configurability
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
The reproduction is available at .
`repro.bzl` defines a minimal test rule and a top-level aspect that emits a file for each target it analyzes. `BUILD.bazel` defines compatible and incompatible tests, a compatible suite, and a suite containing both tests.
First, confirm the aspect can analyze an unexpanded compatible suite:
```sh
$ bazelisk build --noexpand_test_suites --aspects=//:repro.bzl%manifest_aspect --output_groups=repro //:compatible_suite
INFO: Analyzed target //:compatible_suite (3 packages loaded, 50 targets configured, 2 aspect applications).
INFO: Found 1 target...
Aspect //:repro.bzl%manifest_aspect of //:compatible_suite up-to-date:
bazel-bin/compatible_suite.aspect.txt
```
Then build the mixed suite with the same aspect:
```sh
$ bazelisk build --noexpand_test_suites --aspects=//:repro.bzl%manifest_aspect --output_groups=repro //:mixed_suite
ERROR: Analysis of target '//:mixed_suite' failed; build aborted: Target //:mixed_suite is incompatible and cannot be built, but was explicitly requested.
Dependency chain:
//:mixed_suite (e722a2)
//:incompatible_test (e722a2) <-- target platform (@@platforms//host:host) didn't satisfy constraint @@platforms//:incompatible
ERROR: Build did NOT complete successfully
```
Adding `--skip_incompatible_explicit_targets`:
```sh
$ bazelisk build --noexpand_test_suites --skip_incompatible_explicit_targets --aspects=//:repro.bzl%manifest_aspect --output_groups=repro //:mixed_suite
INFO: Analyzed target //:mixed_suite (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:mixed_suite was skipped
INFO: Build completed successfully, 1 total action
```
Leaving test-suite expansion enabled:
```sh
$ bazelisk build --aspects=//:repro.bzl%manifest_aspect --output_groups=repro //:mixed_suite
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured, 1 aspect application).
INFO: Found 2 targets...
INFO: Build completed successfully, 2 total actions
```
### Which operating system are you running Bazel on?
AlmaLinux 9.7 (Moss Jungle Cat), x86_64.
### What is the output of `bazel info release`?
release 9.2.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
N/A, reproduced with the Bazel 9.2.0 release binary.
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
The failure reproduces from Bazel 4.0.0 onward.
### Have you found anything relevant by searching the web?
- #29137 is about a `test_suite` target's own `target_compatible_with` being ignored during normal suite expansion. This issue is about incompatibility inherited from a suite member when expansion is disabled.
- #18707 and https://github.com/bazelbuild/bazel/discussions/19200 discuss target incompatibility inheritance in general. This issue is specifically about its interaction with `--noexpand_test_suites` and top-level aspects.
- #11455 requests configurable `test_suite.tests`, which isn't required here.
### Any other information, logs, or outputs that you want to share?
The `--expand_test_suites` help text in Bazel 9.2.0 says:
```text
--[no]expand_test_suites (a boolean; default: "true")
Expand test_suite targets into their constituent tests before analysis.
When this flag is turned on (the default), negative target patterns will
apply to the tests belonging to the test suite, otherwise they will not.
Turning off this flag is useful when top-level aspects are applied at
command line: then they can analyze test_suite targets.
Tags: loading_and_analysis
```
Contributor guide
Research direction
Start by running the commands from the issue against the repro.bzl and BUILD.bazel example, comparing --noexpand_test_suites with expansion enabled and with --skip_incompatible_explicit_targets. Trace how the mixed test suite becomes incompatible before the top-level aspect is applied. Done means the intended suite-level aspect behavior is defined and the reproduction no longer prevents compatible members from being analyzed.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100