apache / apache/maven-surefire
[Regression] Behavior change in TestNG extension's exception handling breaks builds
- Dominant language
- Java
- Stars
- 461
- Forks
- 588
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 19
Description
### Affected version
3.6.0
### Bug description
Starting with 3.6.0, when TestNG extension's `onStart()` (and maybe `onFinish()`) methods would throw an exception,
a generic error `TestEngine with ID 'testng' failed to discover tests` is reported by the build.
This error doesn't point to the actual root cause in any way, which makes these types of issues very hard, if not impossible.
to debug.
Previous to 3.6.0, these exceptions would be swallowed. Arguably, this was also incorrect behavior.
My proposal would be to report true error, with the stack trace of the root cause, in this particular case, it would be the exception thrown by the reproducer inside `MyTestNGExtension` class.
Reproducer: https://github.com/lprimak/maven-test-compiler-reproducer
`./mvnw package` leads to the following exception
```
[INFO] --- surefire:3.6.0:test (default-test) @ maven-testng-reproducer ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] TestNG is present. Adding per default org.junit.support:testng-engine:1.1.0
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
MyTestNGExtension: onStart called for suite: Command line suite
[ERROR] TestEngine with ID 'testng' failed to discover tests
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.612 s
[INFO] Finished at: 2026-09-07T19:09:17-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.6.0:test (default-test) on project maven-testng-reproducer:
[ERROR]
[ERROR] See /Users/lprimak/Downloads/maven-testng-reproducer/target/surefire-reports for the individual test results.
[ERROR] See dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] There was an error in the forked process
[ERROR] TestEngine with ID 'testng' failed to discover tests
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
[ERROR] TestEngine with ID 'testng' failed to discover tests
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:708)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:523)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:323)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:287)
[ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1474)
[ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1246)
[ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:1082)
[ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162)
[ERROR] at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
[ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159)
[ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)
[ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73)
[ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53)
[ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173)
[ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101)
[ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:919)
[ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:285)
[ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:207)
[ERROR] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
[ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:565)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:255)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:201)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:362)
[ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:314)
```
Previously, no error would occur:
`./mvnw package -Pprevious-testng`
```
[INFO] --- surefire:3.5.6:test (default-test) @ maven-testng-reproducer ---
[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
MyTestNGExtension: onStart called for suite: Surefire suite
MyTestNGExtension: onFinish called for suite: Surefire suite
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.131 s -- in TestSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
```
The issue is reproducible via maven 3.9.16 and 4-rc-6
Contributor guide
Research direction
Start by running ./mvnw package against the linked maven-test-compiler-reproducer and compare it with ./mvnw package -Pprevious-testng. Trace the TestNG extension onStart/onFinish exception path and the TestNG engine discovery failure; done means the original exception and stack trace are reported instead of only the generic discovery error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100