apache / apache/maven-surefire

[SUREFIRE-1477] CI support for parallel test execution

Open
#2,003 1 comment 0 reactions 0 assignees View on GitHub
enhancement priority:major
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

**[Stephen Connolly](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stephenc)** opened **[SUREFIRE-1477](https://issues.apache.org/jira/browse/SUREFIRE-1477?redirect=false)** and commented

CI build servers often want to use multiple compute resources to reduce build times in exchange for additional CPU costs.

One of the biggest consumers of build time is test execution, so much so that people are often willing to run multiple builds in parallel on different machines in order to get the total testing wall clock time lower.

For example: https://wiki.jenkins.io/display/JENKINS/Parallel+Test+Executor+Plugin

It would be nice if surefire made all this easier and more CI system agnostic.

We already have some of this available: https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#runOrder has the option `balanced` that will use a statistics file `.surefire-XXXXXXXXX` to distribute tests between different parallel worker threads.

What is needed to make this better is the ability to use the `balanced` run ordering across Maven JVMs.

If we had two system properties:
* `surefire.mvnCount`
* `surefire.mvnOffset`

Then surefire could use the `balanced` run order to deterministically decide the ordered list of tests... it would then start at `mvnOffset` and run every `mvnCount`th test.

CI users could then run three maven builds in parallel:
* `mvn -Dsurefire.mvnCount=3 -Dsurefire.mvnOffset=0 test`
* `mvn -Dsurefire.mvnCount=3 -Dsurefire.mvnOffset=1 test`
* `mvn -Dsurefire.mvnCount=3 -Dsurefire.mvnOffset=2 test`

This would ensure that all the tests are run once, but the wall clock time should be best case approx 1/3th of the wall clock for the full build time

 

---

**Issue Links:**
- [SUREFIRE-1529](https://issues.apache.org/jira/browse/SUREFIRE-1529) Implement test bucketing
(_**"is duplicated by"**_)

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked SUREFIRE-1529 issue, which is marked as a duplicate, and then inspect the existing balanced run-order behavior described in the Maven Surefire documentation. Use the three example Maven commands as the intended CI scenarios; done means tests are deterministically distributed across Maven JVMs without omissions or overlap.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.