jenkinsci / jenkinsci/parallel-test-executor-plugin
Missing time estimation for tests that have not been executed in the last build
- Dominant language
- Java
- Stars
- 54
- Forks
- 58
- PR merge metrics
- No merged PRs in 30d
Description
I have the following scenario: some tests for a project A is flaky on the CI but passes when executed locally. As executing the whole test suite takes several hours, I wanted to execute just the flaky tests. So I made some modifications to the pom and created a pull-request (PR1). Some time later I investigated a timeout for another pull-request (PR2) and discovered that it reached timeout because it executed 2 splits instead of the usual 20 splits.
After some investigation, I came to the following conclusion: PR2 used the build of PR1 as a reference to estimate the execution times of the tests (`Using build as reference` in the logs). However, as PR1 didn't execute most of them, the [Splitter](https://github.com/jenkinsci/parallel-test-executor-plugin/blob/23a9285adc839c8d7527d6de12a118b80da1112f/src/main/java/org/jenkinsci/plugins/parallel_test_executor/Splitter.java#L89) called one of the Parallelism implementations with a much shorter list of tests, resulting in a much lower number of splits.
**Suggestion**
Which one of the following two approaches would be better? Or are there other alternatives?
* If the time for a test could not be estimated using the last build, the Splitter should look into the earlier builds and try to get a time estimation from them.
* It might be worthwhile to consider checking the previous build of the target branch first, to avoid that one PR influences the subsequent PRs:
https://github.com/jenkinsci/parallel-test-executor-plugin/blob/23a9285adc839c8d7527d6de12a118b80da1112f/src/main/java/org/jenkinsci/plugins/parallel_test_executor/Splitter.java#L199-L201
**Related tickets / PRs:**
* https://github.com/jenkinsci/parallel-test-executor-plugin/issues/83
* https://github.com/jenkinsci/parallel-test-executor-plugin/pull/26
* https://github.com/jenkinsci/parallel-test-executor-plugin/pull/34
**Addendum**
Just for the record, to limit the execution to the test that I wanted to run, I've made changes to project A's pom.xml analogous to the following:
```
org.apache.maven.plugins
maven-surefire-plugin
${maven.surefire.plugin.version}
TestToInvestigate
**/*.java
**/TestToInvestigate.java
```
Contributor guide
Assessment
This issue has not been assessed yet.