apache / apache/maven-surefire

[SUREFIRE-1387] runTime always 0 in statistics, breaking runOrder=balanced

Open
#2,301 5 comments 0 reactions 0 assignees View on GitHub
bug priority:major
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

**[Emond Papegaaij](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=papegaaij)** opened **[SUREFIRE-1387](https://issues.apache.org/jira/browse/SUREFIRE-1387?redirect=false)** and commented

We are using parallized tests to speed up the builds, however tests are executed in less than optimal order. `balanced` is supposed to improve the ordering, but it requires statistics in a `.surefire-XXXXX` file. A maven build does write this file, but runTime is reported as 0 for all tests. Every line in the file looks like (where n is the number of executions):

```
n,0,method(class)
```

I've traced the source of these '0's to nulls in the `ForkedBooter` protocol. These are written by `ForkingRunListener.testSucceeded` on this stack:

```
Thread [main] (Suspended (breakpoint at line 151 in ForkingRunListener))
owns: Notifier (id=14012)
ForkingRunListener.testSucceeded(ReportEntry) line: 151
JUnit4RunListener.testFinished(Description) line: 140
SynchronizedRunListener.testFinished(Description) line: 56
RunNotifier$7.notifyListener(RunListener) line: 190
RunNotifier$7(RunNotifier$SafeNotifier).run() line: 72
Notifier(RunNotifier).fireTestFinished(Description) line: 187
JUnitSupervisor.afterFeature(FeatureInfo) line: 193
ParameterizedSpecRunner(BaseSpecRunner).runFeature() line: 239
ParameterizedSpecRunner(BaseSpecRunner).runFeatures() line: 188
ParameterizedSpecRunner(BaseSpecRunner).doRunSpec() line: 98
BaseSpecRunner$1.invoke(Object, Object...) line: 84
ParameterizedSpecRunner(BaseSpecRunner).invokeRaw(Object, MethodInfo, Object...) line: 481
ParameterizedSpecRunner(BaseSpecRunner).invoke(Object, MethodInfo, Object...) line: 464
ParameterizedSpecRunner(BaseSpecRunner).runSpec() line: 76
ParameterizedSpecRunner(BaseSpecRunner).run() line: 67
ArquillianSputnik.run(RunNotifier) line: 135
JUnit4Provider.execute(Class, Notifier, Filter) line: 365
JUnit4Provider.executeWithRerun(Class, Notifier) line: 272
JUnit4Provider.executeTestSet(Class, RunListener, Notifier) line: 236
JUnit4Provider.invoke(Object) line: 159
ForkedBooter.invokeProviderInSameClassLoader(Object, Object, ProviderConfiguration, boolean, StartupConfiguration, boolean) line: 386
ForkedBooter.runSuitesInProcess(Object, StartupConfiguration, ProviderConfiguration, PrintStream) line: 323
ForkedBooter.main(String...) line: 143
```

`JUnit4RunListener.testFinished(Description)` always creates TestReports with 'elapsed == null'.

We are using the following configuration:

```

3
true
balanced
false

```

---

**Affects:** 2.20

1 votes, 3 watchers

Contributor guide

Open the contributing guide

Research direction

Start with JUnit4RunListener.testFinished(Description), where TestReports are created with elapsed == null, then follow the value into ForkingRunListener.testSucceeded(ReportEntry) and the ForkedBooter protocol. Check the existing statistics output for the .surefire-XXXXX file and its runTime field; done means completed tests record their elapsed time instead of zero so balanced ordering has usable statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.