[SUREFIRE-1638] untilFailureLoopCount property to run tests until failure

**[Galder Zamarreño](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=zkxnca)** opened **[SUREFIRE-1638](https://issues.apache.org/jira/browse/SUREFIRE-1638?redirect=false)** and commented

This is a proposal to add surefire/failsafe property called `untilFailureLoopCount`.

I often find myself in the situation where there's a random test failure happening in CI, and have to run the one or several tests continuously until it fails, maybe running them with TRACE logging.

Normally, I'd wrap the `mvn test...` call with a script that calls it in a loop, but this can be quite "slow" since each iteration is a new JVM launch...etc.

The idea behind `untilFailureLoopCount` is that you've give it a positive number, say 1000, and the test(s) would run continuously until either a failure happens or the test is executed that many times.

I've created [this randomly failing test](https://github.com/galderz/surefire-until-failure-loop-count/blob/master/src/test/java/com/acme/maven/surefire/untilfailure/UntilFailureLoopCountTest.java) and I've run it with `mvn -Dsurefire.untilFailureLoopCount=10 test` and the output would be something like:

```
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.034 s - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.001 s <<< FAILURE! - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[ERROR] com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest.testEventualFailure Time elapsed: 0.001 s <<< FAILURE!
java.lang.AssertionError: Just messing with your testing
at com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest.testEventualFailure(UntilFailureLoopCountTest.java:16)
```

If maintainers are interested in this, I can complete the PR with tests, documentation and support for other providers.

---

1 votes, 3 watchers

Start by reviewing the linked randomly failing UntilFailureLoopCountTest.java example and the current Surefire/Failsafe test execution behavior. Run the demonstrated Maven command to understand the expected output. Done means implementing the proposed property with tests, documentation, and support for the other providers described in the issue.

Written by the indexing model from the issue text.