jenkinsci / jenkinsci/parallel-test-executor-plugin

Missing time estimation for tests that have not been executed in the last build

Open
#354 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
54
Forks
58
PR merge metrics
No merged PRs in 30d

Description

I have the following scenario: some tests for a project A is flaky on the CI but passes when executed locally. As executing the whole test suite takes several hours, I wanted to execute just the flaky tests. So I made some modifications to the pom and created a pull-request (PR1). Some time later I investigated a timeout for another pull-request (PR2) and discovered that it reached timeout because it executed 2 splits instead of the usual 20 splits.

After some investigation, I came to the following conclusion: PR2 used the build of PR1 as a reference to estimate the execution times of the tests (`Using build as reference` in the logs). However, as PR1 didn't execute most of them, the [Splitter](https://github.com/jenkinsci/parallel-test-executor-plugin/blob/23a9285adc839c8d7527d6de12a118b80da1112f/src/main/java/org/jenkinsci/plugins/parallel_test_executor/Splitter.java#L89) called one of the Parallelism implementations with a much shorter list of tests, resulting in a much lower number of splits.

**Suggestion**
Which one of the following two approaches would be better? Or are there other alternatives?

* If the time for a test could not be estimated using the last build, the Splitter should look into the earlier builds and try to get a time estimation from them.

* It might be worthwhile to consider checking the previous build of the target branch first, to avoid that one PR influences the subsequent PRs:

https://github.com/jenkinsci/parallel-test-executor-plugin/blob/23a9285adc839c8d7527d6de12a118b80da1112f/src/main/java/org/jenkinsci/plugins/parallel_test_executor/Splitter.java#L199-L201

**Related tickets / PRs:**
* https://github.com/jenkinsci/parallel-test-executor-plugin/issues/83
* https://github.com/jenkinsci/parallel-test-executor-plugin/pull/26
* https://github.com/jenkinsci/parallel-test-executor-plugin/pull/34

**Addendum**
Just for the record, to limit the execution to the test that I wanted to run, I've made changes to project A's pom.xml analogous to the following:
```

org.apache.maven.plugins
maven-surefire-plugin
${maven.surefire.plugin.version}

TestToInvestigate

**/*.java

**/TestToInvestigate.java



```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.