apache / apache/maven-surefire

[SUREFIRE-2041] Order test classes according to -Dtest

Open
#2,603 0 comments 0 reactions 0 assignees View on GitHub
enhancement priority:major
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

**[Aslak Hellesøy](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=JIRAUSER286867)** opened **[SUREFIRE-2041](https://issues.apache.org/jira/browse/SUREFIRE-2041?redirect=false)** and commented

I'm working on a Test Case Prioritization tool that uses machine learning to predict what tests are most likely to fail (based on historic changes and test results). The output from this tool is a list of test names, ordered by decreasing probability of failure. This tool can provide the order as test classes or as test methods. For example:

Class order:
* testing.Y
* testing.X

Method order:
* testing.Y#a
* testing.X#c
* testing.Y#b

Ideally, it would be possible to launch surefire with a test order that uses a list of tests to indicate what order they should be run in, e.g:

`# class order`
`mvn test -Dsurefire.runOrder=test -Dtest=testing.Y,testing.X`

`# method order`
`mvn test -Dsurefire.runOrder=test -Dtest=testing.Y#a,testing.X#c,testing.Y#b`

From what I can tell, the current design of Surefire wouldn't allow the kind of method ordering illustrated above. It cannot run {**}testing.Y#a{**}, then {**}testing.X#c{**}, then {**}testing.Y#b{**}. There is an inherent assumptions in **TestsToRun** that the overall ordering is by class.

I would therefore be ok with just class ordering for now.

Previous relates issues and pull requests:
* https://issues.apache.org/jira/browse/SUREFIRE-1405
* https://github.com/apache/maven-surefire/pull/169
* https://github.com/apache/maven-surefire/pull/348

---

**Remote Links:**
- [GitHub Pull Request #495
](https://github.com/apache/maven-surefire/pull/495)
- [GitHub Pull Request #560
](https://github.com/apache/maven-surefire/pull/560)

Contributor guide

Open the contributing guide

Research direction

Start by reading the TestsToRun design and the earlier SUREFIRE-1405 issue and pull requests #169 and #348. Compare the requested class-order commands with the current test execution model, then verify that the selected classes run in the order supplied through -Dtest and -Dsurefire.runOrder=test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.