apache / apache/maven-surefire

[SUREFIRE-2151] Inconsistent console reporter output on failures for parameterized tests, with/without rerunFailingTestsCount

Open
#2,767 0 comments 0 reactions 0 assignees View on GitHub
bug priority:major
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

**[Ralph Weires](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=rweires)** opened **[SUREFIRE-2151](https://issues.apache.org/jira/browse/SUREFIRE-2151?redirect=false)** and commented

The way in which test-failures are being displayed with the console-reporter is not ideal and partly inconsistent, in particular for (e.g. JUnit5) parameterized tests.

Taking a small (JUnit5) snippet of a dummy-test as example:

```java
public class DummyTest {
 @ParameterizedTest
  @CsvSource({"yes", "no", "yes", "yes", "no"})
  public void dummyTest(String param) {
    testInternal(param);
  }

private void testInternal(String arg) {
    if (arg.equals("no")) {
      Assertions.fail("If you say 'no', it's a no");
    }
  }
}
```

Running this with surefire will display an error like this (the summary in the end):

```java
[...]

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   DummyTest.dummyTest:16->testInternal:21 If you say 'no', it's a no
[ERROR]   DummyTest.dummyTest:16->testInternal:21 If you say 'no', it's a no
[INFO]
[ERROR] Tests run: 5, Failures: 2, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

[...]
```

The failures do show parts of the problematic code-path, but don't have any information about the actual invocations of the parameterized tests that failed (in the example, invocations 2+5 of the 5). And while it is possible to see more details in the stack traces (i.e. scrolling up in the output), it would be quite nice see more details right away.

If _rerunFailingTestsCount_ is used (here with value 2), the output does show more details right away - namely the actual problematic invocations:

```java
[...]

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] test.DummyTest.dummyTest(String)[2]
[ERROR]   Run 1: DummyTest.dummyTest:16->testInternal:21 If you say 'no', it's a no
[ERROR]   Run 2: DummyTest.dummyTest:16->testInternal:21 If you say 'no', it's a no
[ERROR]   Run 3: DummyTest.dummyTest:16->testInternal:21 If you say 'no', it's a no
[INFO]
[ERROR] test.DummyTest.dummyTest(String)[5]
[ERROR]   Run 1: DummyTest.dummyTest:16->testInternal:21 If you say 'no', it's a no
[ERROR]   Run 2: DummyTest.dummyTest:16->testInternal:21 If you say 'no', it's a no
[ERROR]   Run 3: DummyTest.dummyTest:16->testInternal:21 If you say 'no', it's a no
[INFO]
[INFO]
[ERROR] Tests run: 5, Failures: 2, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

[...]
```

In fact, this is currently the main reason for us to even use the _rerunFailingTestsCount_ flag - regardless of what that flag is actually meant for - which feels rather weird.

 

Would it make sense to align this somehow?

---

**Affects:** 3.0.0-M9

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the JUnit5 parameterized-test example with the console reporter, both with and without rerunFailingTestsCount. Trace the console-reporter failure summary and parameterized invocation handling; done means failures identify the problematic invocations consistently without requiring the rerun flag.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.