apache / apache/maven-surefire

[SUREFIRE-1627] Surefire report contains wrong totals on number of tests and errors when used with rerunFailingTestsCount flag

Open
#2,138 2 comments 0 reactions 0 assignees View on GitHub
bug priority:major
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

**[Alex Filatau](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=alex_filatau)** opened **[SUREFIRE-1627](https://issues.apache.org/jira/browse/SUREFIRE-1627?redirect=false)** and commented

Steps:
1. Make sure you have at least one Test class with at least one test method
2. Make this test method be constantly failing throwing an `AssertionError`
3. Run the test via maven using rerun option: `mvn clean test -Dsurefire.rerunFailingTestsCount=5`
4. Compare build output and xml report generated by Surefire.

Issue: the report ` target/surefire-reports/TEST-mainTest.xml` contains wrong counts on total number of tests and failures (in the top level element):

```xml

```

Please note that there is only 1 element **testcase** in the body (versus 6 in the totals), and only one element **failure** as well (versus 6 in the totals). The report is considering each re-run as a separate test and each **rerunFailure** as a failure count. That leads to inconsistencies when parsing the report since there is mismatch between totals and actual number of tree elements. 

Also it contradicts general reporting logic one can see in the build output:

```java
Results :

Failed tests:
mainTest.failingTest(mainTest)
Run 1: mainTest.failingTest:9 Failing test
Run 2: mainTest.failingTest:9 Failing test
Run 3: mainTest.failingTest:9 Failing test
Run 4: mainTest.failingTest:9 Failing test
Run 5: mainTest.failingTest:9 Failing test
Run 6: mainTest.failingTest:9 Failing test

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

```

 
To sum up, reporting counts not just  **rerunFailure**, but also **flakyFailure** as a new failure, and each rerun as a new test for totals counts.

I see the same issue with Surefire versions 2.22.1 and 3.0.0-M3 as well.

---

**Affects:** 2.19.1

**Attachments:**
- [TEST-mainTest.xml](https://issues.apache.org/jira/secure/attachment/12956019/TEST-mainTest.xml) (_6.72 kB_)

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with `mvn clean test -Dsurefire.rerunFailingTestsCount=5` and compare the build totals with `target/surefire-reports/TEST-mainTest.xml`. Trace the Surefire report generation for rerunFailure and flakyFailure entries; done when the XML totals match its testcase and failure elements while the build output still reports one test and one failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.