eclipse-jdt / eclipse-jdt/eclipse.jdt.core

Jenkins builds create bad test reports

Open
#5,325 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
237
Forks
195
Avg merge
1d 12h
Merged PRs (30d)
47

Description

If you look at test results on jenkins, you'll see all compiler regression tests attributed to test class org.eclipse.jdt.core.tests.compiler.regression.TestAll. Additionally, most tests don't specify the name of the test case.

The problem manifests in the generated file [TEST-org.eclipse.jdt.core.tests.compiler.regression.TestAll.xml](https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-5323/1/artifact/org.eclipse.jdt.core.tests.compiler/target/surefire-reports/TEST-org.eclipse.jdt.core.tests.compiler.regression.TestAll.xml), i.e., it is _not_ a problem of how tests are rendered in the jenkins UI.

Actually, the entire test result file consists of a single `` element.

By contrast, if you export such file from Eclipse's JUnit view it contains one `` element per test class, and `` elements correctly show the method name.

Interestingly, even on Jenkins a few tests at least show the method name. The last test showing a non-empty name is `testNativeLibrariesAreDetectedCaseInsensitive` from test class `org.eclipse.jdt.core.tests.compiler.util.UtilTest`. After this, all test names are blank. Interestingly this test class uses JUnit 4 annotations, while the rest of the suite is at JUnit 3.

Contributor guide

Open the contributing guide

Research direction

Start with the generated TEST-org.eclipse.jdt.core.tests.compiler.regression.TestAll.xml report linked in the issue and inspect how the Jenkins build produces its single testsuite. Compare the JUnit 3 regression suite with the JUnit 4 UtilTest, then verify that the report contains separate test suites and non-empty testcase method names for the affected tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.