apache / apache/maven-surefire

JUnitPlatformProvider reruns successful tests

Open
#847 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

### Affected version

3.5.2

### Bug description

Hello!

maven.surefire.version - 3.5.2
junit.version - 5.12.2

We were using Junit 4 in our project. Then we decided to move to the Junit 5 version, but there are still many Junit 4 tests in our project. We are using maven surefire to run tests. Since we've migrated to Junit 5, surefire is using org.apache.maven.surefire.junitplatform.JUnitPlatformProvider to run tests. It was org.apache.maven.surefire.junit4.JUnit4Provider
while we were using Junit 4 only.

The thing is - now when I'm running tests with rerunFailingTestsCount

mvn surefire:test -B -Dtest=JUnit4Test -Dsurefire.rerunFailingTestsCount=2

if the whole test class fails (in [@afterclass](https://github.com/afterclass) for example) - surefire reruns all test classes, even successful test methods.

With JUnitPlatformProvider:

```
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] JUnit4Test.null
[ERROR] Run 1: JUnit4Test.methodThatFailsWholeClass:35
[ERROR] Run 2: JUnit4Test.methodThatFailsWholeClass:35
[ERROR] Run 3: JUnit4Test.methodThatFailsWholeClass:35
[INFO]
[INFO]
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0
```

With JUnit4Provider:
```

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] JUnit4Test.methodThatFailsWholeClass:35
[INFO]
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0
```

I've prepared sample project where this behaviour can be reproduced - https://github.com/Yahmus/JunitProviderTest
You can run the test class, check the behaviour, then switch the provider by adding

```


org.apache.maven.surefire
surefire-junit4
${maven.surefire.version}


```

to maven-surefire-plugin in pom.xml and run the test class again.

I've found a little bit of a similar issue but it doesn't really help [#4026](https://github.com/junit-team/junit5/discussions/4026)
I don't actually understand. Is this a bug or an expected behaviour?

I've tried to raise the issue in your Jira, but no one accepted my account creation request.

Contributor guide

Open the contributing guide

Research direction

Start with the linked JunitProviderTest sample project and run the stated mvn surefire:test command using JUnitPlatformProvider, then compare it with JUnit4Provider. Read the JUnitPlatformProvider rerun path and use the pom.xml provider configuration as the comparison point. Done means a class-level failure reruns only the failing test rather than successful methods.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.