apache / apache/maven-surefire
JUnitPlatformProvider reruns successful tests
- Dominant language
- Java
- Stars
- 461
- Forks
- 588
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 19
Description
### Affected version
3.5.2
### Bug description
Hello!
maven.surefire.version - 3.5.2
junit.version - 5.12.2
We were using Junit 4 in our project. Then we decided to move to the Junit 5 version, but there are still many Junit 4 tests in our project. We are using maven surefire to run tests. Since we've migrated to Junit 5, surefire is using org.apache.maven.surefire.junitplatform.JUnitPlatformProvider to run tests. It was org.apache.maven.surefire.junit4.JUnit4Provider
while we were using Junit 4 only.
The thing is - now when I'm running tests with rerunFailingTestsCount
mvn surefire:test -B -Dtest=JUnit4Test -Dsurefire.rerunFailingTestsCount=2
if the whole test class fails (in [@afterclass](https://github.com/afterclass) for example) - surefire reruns all test classes, even successful test methods.
With JUnitPlatformProvider:
```
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] JUnit4Test.null
[ERROR] Run 1: JUnit4Test.methodThatFailsWholeClass:35
[ERROR] Run 2: JUnit4Test.methodThatFailsWholeClass:35
[ERROR] Run 3: JUnit4Test.methodThatFailsWholeClass:35
[INFO]
[INFO]
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0
```
With JUnit4Provider:
```
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] JUnit4Test.methodThatFailsWholeClass:35
[INFO]
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0
```
I've prepared sample project where this behaviour can be reproduced - https://github.com/Yahmus/JunitProviderTest
You can run the test class, check the behaviour, then switch the provider by adding
```
org.apache.maven.surefire
surefire-junit4
${maven.surefire.version}
```
to maven-surefire-plugin in pom.xml and run the test class again.
I've found a little bit of a similar issue but it doesn't really help [#4026](https://github.com/junit-team/junit5/discussions/4026)
I don't actually understand. Is this a bug or an expected behaviour?
I've tried to raise the issue in your Jira, but no one accepted my account creation request.
Contributor guide
Research direction
Start with the linked JunitProviderTest sample project and run the stated mvn surefire:test command using JUnitPlatformProvider, then compare it with JUnit4Provider. Read the JUnitPlatformProvider rerun path and use the pom.xml provider configuration as the comparison point. Done means a class-level failure reruns only the failing test rather than successful methods.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100