apache / apache/maven-surefire
[SUREFIRE-2041] Order test classes according to -Dtest
- Dominant language
- Java
- Stars
- 461
- Forks
- 588
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 19
Description
**[Aslak Hellesøy](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=JIRAUSER286867)** opened **[SUREFIRE-2041](https://issues.apache.org/jira/browse/SUREFIRE-2041?redirect=false)** and commented
I'm working on a Test Case Prioritization tool that uses machine learning to predict what tests are most likely to fail (based on historic changes and test results). The output from this tool is a list of test names, ordered by decreasing probability of failure. This tool can provide the order as test classes or as test methods. For example:
Class order:
* testing.Y
* testing.X
Method order:
* testing.Y#a
* testing.X#c
* testing.Y#b
Ideally, it would be possible to launch surefire with a test order that uses a list of tests to indicate what order they should be run in, e.g:
`# class order`
`mvn test -Dsurefire.runOrder=test -Dtest=testing.Y,testing.X`
`# method order`
`mvn test -Dsurefire.runOrder=test -Dtest=testing.Y#a,testing.X#c,testing.Y#b`
From what I can tell, the current design of Surefire wouldn't allow the kind of method ordering illustrated above. It cannot run {**}testing.Y#a{**}, then {**}testing.X#c{**}, then {**}testing.Y#b{**}. There is an inherent assumptions in **TestsToRun** that the overall ordering is by class.
I would therefore be ok with just class ordering for now.
Previous relates issues and pull requests:
* https://issues.apache.org/jira/browse/SUREFIRE-1405
* https://github.com/apache/maven-surefire/pull/169
* https://github.com/apache/maven-surefire/pull/348
---
**Remote Links:**
- [GitHub Pull Request #495
](https://github.com/apache/maven-surefire/pull/495)
- [GitHub Pull Request #560
](https://github.com/apache/maven-surefire/pull/560)
Contributor guide
Research direction
Start by reading the TestsToRun design and the earlier SUREFIRE-1405 issue and pull requests #169 and #348. Compare the requested class-order commands with the current test execution model, then verify that the selected classes run in the order supplied through -Dtest and -Dsurefire.runOrder=test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100