apache / apache/maven-surefire

[SUREFIRE-2102] Different Behaviour while using junit-jupiter-engine/jupiter-vintage-engine/testng-engine

Open
#2,745 0 comments 0 reactions 0 assignees View on GitHub
bug priority:minor
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

**[Karl Heinz Marbaise](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=khmarbaise)** opened **[SUREFIRE-2102](https://issues.apache.org/jira/browse/SUREFIRE-2102?redirect=false)** and commented

I have a full working example which executes three different kind of tests:

* Junit Jupiter based
* JUnit 4 based
* TestNG based

what I can observe there seemed to be a difference in behaviour.

```
[INFO] Running com.soebes.youtube.maven.episodes.ep3.jupiter.FractionJUnitJupiterTest
FractionJUnitJupiterTest.beforeAll
FractionJUnitJupiterTest.beforeEach
FractionJUnitJupiterTest.first_add
FractionJUnitJupiterTest.afterEach
FractionJUnitJupiterTest.afterAll
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 s - in com.soebes.youtube.maven.episodes.ep3.jupiter.FractionJUnitJupiterTest
[INFO] Running com.soebes.youtube.maven.episodes.ep3.junit.FractionJUnit4Test
FractionJUnit4Test.beforeClass
FractionJUnit4Test.before
FractionJUnit4Test.first_add
FractionJUnit4Test.after
FractionJUnit4Test.afterClass
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s - in com.soebes.youtube.maven.episodes.ep3.junit.FractionJUnit4Test
[INFO] Running com.soebes.youtube.maven.episodes.ep3.testng.FractionTestNGTest
FractionTestNGTest.beforeClass
FractionTestNGTest.beforeMethod
FractionTestNGTest.first_add
FractionTestNGTest.afterMethod
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in com.soebes.youtube.maven.episodes.ep3.testng.FractionTestNGTest
FractionTestNGTest.afterClass
[DEBUG] Closing the fork 1 after saying GoodBye.
```

You can see that the output of the `afterClass` is after the output of the test result. But only for TestNG while for JUnit 4 and JUnit Jupiter it looks the same.

The same happens for using failsafe in the same example project.

I made a full working reproducible example: https://github.com/khmarbaise/sffs-bugs (Branch: main). This requires JDK 17 (but should also work with JDK11)...

 The repository contains two full DEBUG logoutput files: `mvn.log` (running `mvn verify`) and `mvn-test.log` only for running unit test via `mvn test`

---

**Affects:** 3.0.0-M7

Contributor guide

Open the contributing guide

Research direction

Start with the reproducible example in the sffs-bugs repository and run it with JDK 17 using mvn test and mvn verify. Compare mvn.log and mvn-test.log, focusing on why TestNG afterClass output appears after the test result while the JUnit Jupiter and JUnit 4 output does not; done means the lifecycle output is consistent across engines.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.