eclipse-jdt / eclipse-jdt/eclipse.jdt.core
Jenkins builds create bad test reports
- Dominant language
- Java
- Stars
- 237
- Forks
- 195
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 47
Description
If you look at test results on jenkins, you'll see all compiler regression tests attributed to test class org.eclipse.jdt.core.tests.compiler.regression.TestAll. Additionally, most tests don't specify the name of the test case.
The problem manifests in the generated file [TEST-org.eclipse.jdt.core.tests.compiler.regression.TestAll.xml](https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-5323/1/artifact/org.eclipse.jdt.core.tests.compiler/target/surefire-reports/TEST-org.eclipse.jdt.core.tests.compiler.regression.TestAll.xml), i.e., it is _not_ a problem of how tests are rendered in the jenkins UI.
Actually, the entire test result file consists of a single `` element.
By contrast, if you export such file from Eclipse's JUnit view it contains one `` element per test class, and `` elements correctly show the method name.
Interestingly, even on Jenkins a few tests at least show the method name. The last test showing a non-empty name is `testNativeLibrariesAreDetectedCaseInsensitive` from test class `org.eclipse.jdt.core.tests.compiler.util.UtilTest`. After this, all test names are blank. Interestingly this test class uses JUnit 4 annotations, while the rest of the suite is at JUnit 3.
Contributor guide
Research direction
Start with the generated TEST-org.eclipse.jdt.core.tests.compiler.regression.TestAll.xml report linked in the issue and inspect how the Jenkins build produces its single testsuite. Compare the JUnit 3 regression suite with the JUnit 4 UtilTest, then verify that the report contains separate test suites and non-empty testcase method names for the affected tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100