apache / apache/maven-surefire
[SUREFIRE-1387] runTime always 0 in statistics, breaking runOrder=balanced
- Dominant language
- Java
- Stars
- 461
- Forks
- 588
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 19
Description
**[Emond Papegaaij](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=papegaaij)** opened **[SUREFIRE-1387](https://issues.apache.org/jira/browse/SUREFIRE-1387?redirect=false)** and commented
We are using parallized tests to speed up the builds, however tests are executed in less than optimal order. `balanced` is supposed to improve the ordering, but it requires statistics in a `.surefire-XXXXX` file. A maven build does write this file, but runTime is reported as 0 for all tests. Every line in the file looks like (where n is the number of executions):
```
n,0,method(class)
```
I've traced the source of these '0's to nulls in the `ForkedBooter` protocol. These are written by `ForkingRunListener.testSucceeded` on this stack:
```
Thread [main] (Suspended (breakpoint at line 151 in ForkingRunListener))
owns: Notifier (id=14012)
ForkingRunListener.testSucceeded(ReportEntry) line: 151
JUnit4RunListener.testFinished(Description) line: 140
SynchronizedRunListener.testFinished(Description) line: 56
RunNotifier$7.notifyListener(RunListener) line: 190
RunNotifier$7(RunNotifier$SafeNotifier).run() line: 72
Notifier(RunNotifier).fireTestFinished(Description) line: 187
JUnitSupervisor.afterFeature(FeatureInfo) line: 193
ParameterizedSpecRunner(BaseSpecRunner).runFeature() line: 239
ParameterizedSpecRunner(BaseSpecRunner).runFeatures() line: 188
ParameterizedSpecRunner(BaseSpecRunner).doRunSpec() line: 98
BaseSpecRunner$1.invoke(Object, Object...) line: 84
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 481
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 464
ParameterizedSpecRunner(BaseSpecRunner).runSpec() line: 76
ParameterizedSpecRunner(BaseSpecRunner).run() line: 67
ArquillianSputnik.run(RunNotifier) line: 135
JUnit4Provider.execute(Class, Notifier, Filter) line: 365
JUnit4Provider.executeWithRerun(Class, Notifier) line: 272
JUnit4Provider.executeTestSet(Class, RunListener, Notifier) line: 236
JUnit4Provider.invoke(Object) line: 159
ForkedBooter.invokeProviderInSameClassLoader(Object, Object, ProviderConfiguration, boolean, StartupConfiguration, boolean) line: 386
ForkedBooter.runSuitesInProcess(Object, StartupConfiguration, ProviderConfiguration, PrintStream) line: 323
ForkedBooter.main(String...) line: 143
```
`JUnit4RunListener.testFinished(Description)` always creates TestReports with 'elapsed == null'.
We are using the following configuration:
```
3
true
balanced
false
```
---
**Affects:** 2.20
1 votes, 3 watchers
Contributor guide
Research direction
Start with JUnit4RunListener.testFinished(Description), where TestReports are created with elapsed == null, then follow the value into ForkingRunListener.testSucceeded(ReportEntry) and the ForkedBooter protocol. Check the existing statistics output for the .surefire-XXXXX file and its runTime field; done means completed tests record their elapsed time instead of zero so balanced ordering has usable statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100