apache / apache/maven-surefire

3.6.0 regression: Spock specs abort discovery (Tests run: 0) when `<excludedGroups>` is set and JUnit 4 is on the test classpath

Open
#3,468 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

### Affected version

3.6.0 and 3.6.0-M1. Works with 3.5.6.

### Bug description

> *Drafted with AI assistance. I built and ran the reproducer and verified the analysis myself.*

A Spock specification aborts JUnit Platform discovery when `` is configured and the JUnit 4 jar is on the test classpath. The fork dies before any test runs, so the module reports `Tests run: 0` — plain Jupiter tests in the same module do not run either.

Minimal reproducer: https://github.com/MenschNestor/surefire-spock-repro (one Spock feature, one Jupiter test)

```
mvn test -Dsurefire.version=3.5.6 # Tests run: 2, BUILD SUCCESS
mvn test -Dsurefire.version=3.6.0 # Tests run: 0, BUILD FAILURE
```

```
[INFO] --- surefire:3.6.0:test (default-test) @ surefire-spock-repro ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[ERROR] Could not find method with name [addition works] in class [com.example.ExampleSpec].
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[ERROR] There was an error in the forked process
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
```

All three conditions are jointly necessary:

| surefire | `excludedGroups` | `junit:junit` on classpath | tests run |
| --- | --- | --- | ---: |
| 3.5.6 | `slow` | yes | 2 |
| 3.6.0 | *(empty)* | yes | 2 |
| 3.6.0 | `slow` | no | 2 |
| 3.6.0 | `slow` | yes | **0, BUILD FAILURE** |
| 3.6.0-M1 | `slow` | yes | **0, BUILD FAILURE** |

## Cause

In `JUnitPlatformProvider#newFilters`, the exclude-category filter is added whenever `org.junit.experimental.categories.Category` is loadable:

```java
if (!useTestNG) {
Optional> categoryClass = getCategoryClass();
if (categoryClass.isPresent()) {
getPropertiesList(EXCLUDEDGROUPS_PROP)
.map(strings -> getExcludeCategoryFilter(strings, categoryClass))
.ifPresent(filters::add);
}
}
```

Unlike the include branch just above it, this is not gated on `JUNIT_VINTAGE_DETECTED`, so having the JUnit 4 jar anywhere on the test classpath is enough — the Vintage engine does not have to be present, and in the reproducer it is not. The resulting `PostDiscoveryFilter` is applied to descriptors from every engine, and to read the `@Category` annotation it calls

```java
methodSource.get().getJavaMethod()
```

in `getExcludeCategoryFilter`, and identically in `getIncludeCategoryFilter`.

Spock's `MethodSource` carries the *feature name*, not a JVM method name: Spock renames feature methods to `$spock_feature_N_M` and keeps the display name in `@FeatureMetadata`. In the reproducer the feature is `def "addition works"()`, so the reported name contains a space and can never resolve reflectively. `MethodSource#getJavaMethod` throws `PreconditionViolationException`, which escapes discovery and takes the fork down.

3.5.6 mapped `groups`/`excludedGroups` to `TagFilter.includeTags`/`excludeTags` only, with no reflection, so a `MethodSource` that does not correspond to a real method was harmless.

This is not specific to Spock. Any JUnit Platform engine whose `MethodSource` does not map one-to-one onto a reflective `Method` will hit it.

## Possible fixes

Gating both category filters on `JUNIT_VINTAGE_DETECTED` would confine them to descriptors that can actually carry `@Category`. Independently, resolving the method defensively — treating an unresolvable `MethodSource` as "no category" rather than letting the exception escape — would stop a discovery-time reflection failure from killing the entire fork.

## Impact

In our multi-module build this turns every module containing a Spock specification into `Tests run: 0` plus a hard failure — five modules.

Contributor guide

Open the contributing guide

Research direction

Start at JUnitPlatformProvider#newFilters and the getIncludeCategoryFilter/getExcludeCategoryFilter paths described in the issue. Run the linked reproducer with Surefire 3.5.6 and 3.6.0, then verify that excludedGroups with JUnit 4 present no longer aborts discovery and that both Spock and Jupiter tests run.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.