TEST TARGETS SUMMARY does not capture failed test and resulted false positive test build
- Dominant language
- Java
- Stars
- 22
- Forks
- 114
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 8
Description
`runtime-api-java_security_0` failed but is not in `TEST TARGETS SUMMARY`. The test build finished in SUCCESS, which is a false positive.
related: [internal issue](https://github.ibm.com/runtimes/backlog/issues/1617)
```
08:09:01 Test results: passed: 448; failed: 1
...
08:09:02 runtime-api-java_security_0_FAILED
```
```
10:08:39 TEST TARGETS SUMMARY
10:08:39 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10:08:39 DISABLED test targets:
10:08:39 compiler-api-javax_tools_0
10:08:39 runtime-api-java_util-regex_0
10:08:39
10:08:39 PASSED test targets:
10:08:39 compiler-api-signaturetest_0 - Test results: passed: 2
10:08:39 runtime-api-java_lang-instrument_0 - Test results: passed: 10
10:08:39 runtime-api-java_lang-invoke_0 - Test results: passed: 110
10:08:39 runtime-api-java_lang-reflect_0 - Test results: passed: 143
10:08:39 runtime-api-java_lang-Package_0 - Test results: passed: 12
10:08:39 runtime-api-java_lang-String_0 - Test results: passed: 64
10:08:39 runtime-api-java_lang-StringBuilder_0 - Test results: passed: 25
10:08:39 runtime-api-java_lang-StringBuffer_0 - Test results: passed: 28
10:08:39 runtime-api-java_lang-ClassLoader_0 - Test results: passed: 35
10:08:39 runtime-api-java_lang-ModuleLayer_0 - Test results: passed: 23
10:08:39 runtime-api-java_lang-module_0 - Test results: passed: 163
10:08:39 runtime-api-java_lang-StackWalker_0 - Test results: passed: 8
10:08:39 runtime-api-java_lang-Module_class_0 - Test results: passed: 42
10:08:39 runtime-api-java_lang-SecurityManager_0 - Test results: passed: 2
10:08:39 runtime-api-java_lang-System_0 - Test results: passed: 26
10:08:39 runtime-api-java_lang-Class_0 - Test results: passed: 83
10:08:39 runtime-api-signaturetest_0 - Test results: passed: 20
10:08:39 runtime-api-modulegraph_0 - Test results: passed: 1
10:08:39 runtime-api-javax_naming-spi_0 - Test results: passed: 129
10:08:39
10:08:39 TOTAL: 21 EXECUTED: 19 PASSED: 19 FAILED: 0 DISABLED: 2 SKIPPED: 0
10:08:39 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10:08:39
10:08:39 TESTCASES RESULTS SUMMARY: passed: 32,585; failed: 1; error: 0; skipped: 0
10:08:39 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10:08:39 ALL TESTS PASSED
...
Finished: SUCCESS
```
Contributor guide
Research direction
Start by reproducing the mismatch between the failed runtime-api-java_security_0 result and the TEST TARGETS SUMMARY, then trace the reporting path that produces TESTCASES RESULTS SUMMARY and the final build status. Done means failed targets appear in the summary and a failed test cannot produce ALL TESTS PASSED or Finished: SUCCESS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100