apache / apache/maven-surefire

[Regression] TestNG extension onStart/onFinish called too many times

Open
#3,462 4 comments 0 reactions 1 assignee Claimed by @olamy View on GitHub
bug
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

### Affected version

3.6.0

### Bug description

Starting with 3.6.0, TestNG extension `onStart()` and `onFinish()` methods get called too many times.
Previous plugin versions would only call those methods one time.
There is one more caveat to this. If only JUnit Jupiter exist (No TestNG tests), but TestNG is on the Classpath,
TestNG extension should not be invoked at all. This is how previous versions would work.
Here, `onStart()` gets called when there are no tests, which was breaking one of my builds.

Reproducer: https://github.com/lprimak/maven-test-compiler-reproducer
`./mvnw package -Dexception.from.testng=false`

Messages message:
```
[INFO] --- surefire:3.6.0:test (default-test) @ maven-testng-reproducer ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] TestNG is present. Adding per default org.junit.support:testng-engine:1.1.0
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
MyTestNGExtension: onStart called for suite: Command line suite
MyTestNGExtension: onFinish called for suite: Command line suite
MyTestNGExtension: onStart called for suite: Command line suite
MyTestNGExtension: onFinish called for suite: Command line suite
MyTestNGExtension: onStart called for suite: Command line suite
MyTestNGExtension: onFinish called for suite: Command line suite
[INFO] Running com.flowlogix.maven.jupiter.JupiterTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 s -- in com.flowlogix.maven.jupiter.JupiterTest
MyTestNGExtension: onStart called for suite: Command line suite
[INFO] Running com.flowlogix.maven.testng.NextGenTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in com.flowlogix.maven.testng.NextGenTest
MyTestNGExtension: onFinish called for suite: Command line suite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
```

The methods should only be called one time, as with the previous plugin:
`./mvnw package -Pprevious-testng`
```
[INFO] --- surefire:3.5.6:test (default-test) @ maven-testng-reproducer ---
[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
MyTestNGExtension: onStart called for suite: Surefire suite
MyTestNGExtension: onFinish called for suite: Surefire suite
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.141 s -- in TestSuite
```

No TestNG extension's methods should be called, as with the previous plugin, since no TestNG tests get found or executed:
`./mvnw package -Pprevious-plugins`
```
[INFO] --- surefire:3.5.6:test (default-test) @ maven-testng-reproducer ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.flowlogix.maven.jupiter.JupiterTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 s -- in com.flowlogix.maven.jupiter.JupiterTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
```

The issue is reproducible via maven 3.9.16 and 4-rc-6

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.