apache / apache/maven-surefire
[SUREFIRE-1219] skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount
- Dominant language
- Java
- Stars
- 461
- Forks
- 588
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 19
Description
**[Sean Flanigan](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sflanigan)** opened **[SUREFIRE-1219](https://issues.apache.org/jira/browse/SUREFIRE-1219?redirect=false)** and commented
According to https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html "failed tests within re-run phase are not included in skipAfterFailureCount". For instance, if skipAfterFailureCount is 1, but every test passes on the second attempt, the tests are "flaky" but not "failures". This should be reflected in summaries, in XML result files and in fail fast (skipAfterFailureCount), but this is not the case.
If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and skipAfterFailureCount=5:
1. I would expect something like this to be printed on the console:
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10
2. The XML report should list \s but no \.
3. All tests should be run, not just 5 of them, in other words skipAfterFailureCount should not count the flaky tests as part of the failure count.
I have some tests demonstrating the problem, which I will attach to this issue.
---
**Affects:** 2.19.1
**Attachments:**
- [SUREFIRE-1219-tests.zip](https://issues.apache.org/jira/secure/attachment/12783517/SUREFIRE-1219-tests.zip) (_27.25 kB_)
Contributor guide
Research direction
Start with the attached SUREFIRE-1219-tests.zip and reproduce the interaction between rerunFailingTestsCount and skipAfterFailureCount in Maven Surefire. Trace how flaky reruns affect console summaries, XML reports, and fail-fast counting. Done means flaky tests are reported separately, XML contains flakyFailure without failures, and all tests run when reruns pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100