apache / apache/maven-surefire

[SUREFIRE-1644] Identify flaky tests in Surefire report

Open
#2,068 2 comments 0 reactions 0 assignees View on GitHub
enhancement priority:major
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

**[André Carmo](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=atmcarmo)** opened **[SUREFIRE-1644](https://issues.apache.org/jira/browse/SUREFIRE-1644?redirect=false)** and commented

Hi all,

 

There is a feature that allows to rerun failing tests: https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html

This feature outputs to the surefire XML report some elements like flakyFailure, flakyError, and others.

 

However, when generating the surefire report (invoking something like mvn surefire-report:report-only) those XML elements are not used and are not considered for the HTML report. That would be a great improvement.

 

I can try to make a pull request for that, but please give me an overview first on where to look at to try to cut some time and some effort.

 

---

**Issue Links:**
- [SUREFIRE-1751](https://issues.apache.org/jira/browse/SUREFIRE-1751) Surefire report shows flaky tests as failures

- [SUREFIRE-1345](https://issues.apache.org/jira/browse/SUREFIRE-1345) Support flakyFailure and flakyError in TestSuiteXmlParser
(_**"depends upon"**_)

Contributor guide

Open the contributing guide

Research direction

Start with the surefire-report:report-only flow and the TestSuiteXmlParser mentioned in SUREFIRE-1345. Trace how flakyFailure and flakyError in the Surefire XML are handled when producing the HTML report, then verify that flaky tests are represented correctly rather than as ordinary failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.