apache / apache/maven-surefire

[SUREFIRE-1219] skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount

Open
#2,471 5 comments 0 reactions 0 assignees View on GitHub
bug priority:minor
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

**[Sean Flanigan](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sflanigan)** opened **[SUREFIRE-1219](https://issues.apache.org/jira/browse/SUREFIRE-1219?redirect=false)** and commented

According to https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html "failed tests within re-run phase are not included in skipAfterFailureCount". For instance, if skipAfterFailureCount is 1, but every test passes on the second attempt, the tests are "flaky" but not "failures". This should be reflected in summaries, in XML result files and in fail fast (skipAfterFailureCount), but this is not the case.

If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and skipAfterFailureCount=5:

1. I would expect something like this to be printed on the console:
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10

2. The XML report should list \s but no \.

3. All tests should be run, not just 5 of them, in other words skipAfterFailureCount should not count the flaky tests as part of the failure count.

I have some tests demonstrating the problem, which I will attach to this issue.

---

**Affects:** 2.19.1

**Attachments:**
- [SUREFIRE-1219-tests.zip](https://issues.apache.org/jira/secure/attachment/12783517/SUREFIRE-1219-tests.zip) (_27.25 kB_)

Contributor guide

Open the contributing guide

Research direction

Start with the attached SUREFIRE-1219-tests.zip and reproduce the interaction between rerunFailingTestsCount and skipAfterFailureCount in Maven Surefire. Trace how flaky reruns affect console summaries, XML reports, and fail-fast counting. Done means flaky tests are reported separately, XML contains flakyFailure without failures, and all tests run when reruns pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.