[SUREFIRE-2286] Surefire 3.x is ignoring the result of @RunWith-based junit4 tests

**[Thomas Mortagne](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=tmortagne)** opened **[SUREFIRE-2286](https://issues.apache.org/jira/browse/SUREFIRE-2286?redirect=false)** and commented

While more classic junit4 tests are executed and reported fine, it seems that [tests generated through a custom @RunWith](https://github.com/xwiki/xwiki-rendering/blob/xwiki-rendering-16.9.0/xwiki-rendering-syntaxes/xwiki-rendering-syntax-xwiki21/src/test/java/org/xwiki/rendering/internal/xwiki21/XWiki21CompatibilityTest.java) are executed (some logs are produced which can only mean that those tests are indeed executed), but their result does not appear in the Surefire report.

Problem is that it means the build never fail even if one of those tests is actually not passing, we've finally noticed it only after a while...

Downgrading to 2.22.2 make the tests behave as expected.

Note that those test classes are handled properly when adding `org.apache.maven.surefire:surefire-junit47` to the surefire plugin's dependencies, but it causes problems with our junit 5 tests, unfortunately, and we have quite a few modules in which both versions of junit are used (in different test classes, of course).

---

**Affects:** 3.0.0

Start by reproducing the issue with XWiki21CompatibilityTest.java under Surefire 3.x, then compare its report and failure behavior with 2.22.2. Investigate how custom @RunWith JUnit 4 tests are discovered alongside JUnit 5 tests; done means their failures appear in the Surefire report without breaking JUnit 5 execution.

Written by the indexing model from the issue text.