[SUREFIRE-2108] Report NoClassDefFoundError during test discovery as test problem if failIfNoTests is false

**[Gunnar Wagenknecht](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=gunnar)** opened **[SUREFIRE-2108](https://issues.apache.org/jira/browse/SUREFIRE-2108?redirect=false)** and commented

We are running Maven Surefire plug-in in a larger project separately from the main build due to scalability reasons. This may lead top situation when the main build (compile) works successfully and an incomplete test classpath goes unnoticed. In such a case the Maven Surefire plug-in fails completely. Our wish is to never fail the build but report any issues as test failures.

We are using:

```
false
false
true
```

Here is an example of a classpath issue:

```
Caused by: org.apache.maven.surefire.api.util.SurefireReflectionException: org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-vintage' failed to discover tests
at org.apache.maven.surefire.api.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:173)
at org.apache.maven.surefire.api.util.ReflectionUtils.invokeGetter(ReflectionUtils.java:76)
at org.apache.maven.surefire.api.util.ReflectionUtils.invokeGetter(ReflectionUtils.java:70)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.getSuites(ProviderFactory.java:145)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.getSuitesIterator(ForkStarter.java:751)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:351)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:326)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:269)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1332)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1165)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:929)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:140)
... 11 common frames omitted
Caused by: org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-vintage' failed to discover tests
at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:160)
at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverSafely(EngineDiscoveryOrchestrator.java:134)
at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:108)
at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:80)
at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:110)
at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:78)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.discover(DefaultLauncherSession.java:81)
at org.apache.maven.surefire.junitplatform.LazyLauncher.discover(LazyLauncher.java:48)
at org.apache.maven.surefire.junitplatform.TestPlanScannerFilter.accept(TestPlanScannerFilter.java:56)
at org.apache.maven.surefire.api.util.DefaultScanResult.applyFilter(DefaultScanResult.java:102)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.scanClasspath(JUnitPlatformProvider.java:167)
at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.getSuites(JUnitPlatformProvider.java:109)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.maven.surefire.api.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:165)
... 22 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/custommonkey/xmlunit/DifferenceListener
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
at java.base/java.lang.Class.privateGetPublicMethods(Class.java:3191)
at java.base/java.lang.Class.getMethods(Class.java:1904)
at org.junit.platform.commons.util.ReflectionUtils.getDefaultMethods(ReflectionUtils.java:1518)
at org.junit.platform.commons.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:1491)
at org.junit.platform.commons.util.ReflectionUtils.findAllMethodsInHierarchy(ReflectionUtils.java:1433)
at org.junit.platform.commons.util.ReflectionUtils.findMethods(ReflectionUtils.java:1417)
at org.junit.platform.commons.util.ReflectionUtils.findMethods(ReflectionUtils.java:1403)
at org.junit.vintage.engine.descriptor.TestSourceProvider.lambda$findMethod$1(TestSourceProvider.java:75)
at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1134)
at java.base/java.util.Collections$SynchronizedMap.computeIfAbsent(Collections.java:2682)
at org.junit.vintage.engine.descriptor.TestSourceProvider.findMethod(TestSourceProvider.java:75)
at org.junit.vintage.engine.descriptor.TestSourceProvider.computeTestSource(TestSourceProvider.java:56)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
at org.junit.vintage.engine.descriptor.TestSourceProvider.findTestSource(TestSourceProvider.java:47)
at org.junit.vintage.engine.discovery.RunnerTestDescriptorPostProcessor.addChildrenRecursively(RunnerTestDescriptorPostProcessor.java:62)
at org.junit.vintage.engine.discovery.RunnerTestDescriptorPostProcessor.applyFiltersAndCreateDescendants(RunnerTestDescriptorPostProcessor.java:41)
at org.junit.vintage.engine.discovery.VintageDiscoverer.discover(VintageDiscoverer.java:46)
at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:64)
at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:152)
... 38 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.custommonkey.xmlunit.DifferenceListener
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.apache.maven.surefire.booter.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:100)
... 59 common frames omitted
```

I reported the issue also at https://github.com/junit-team/junit5/issues/2971 but I wonder if this can be handled by Maven Surefire directly independent of the test engine.

---

**Affects:** 3.0.0-M6

Start at AbstractSurefireMojo.execute and follow ForkStarter.run through JUnitPlatformProvider.getSuites and scanClasspath, where discovery errors reach the Maven plugin. Check how failIfNoTests, failIfNoSpecifiedTests, and testFailureIgnore are handled. Done means a NoClassDefFoundError during discovery is reported as a test problem without failing the build when the requested options are false or true as described.

Written by the indexing model from the issue text.