adoptium / adoptium/TKG

TEST TARGETS SUMMARY does not capture failed test and resulted false positive test build

Open
#680 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
22
Forks
114
Avg merge
4d 2h
Merged PRs (30d)
8

Description

`runtime-api-java_security_0` failed but is not in `TEST TARGETS SUMMARY`. The test build finished in SUCCESS, which is a false positive.

related: [internal issue](https://github.ibm.com/runtimes/backlog/issues/1617)

```
08:09:01 Test results: passed: 448; failed: 1
...
08:09:02 runtime-api-java_security_0_FAILED
```
```
10:08:39 TEST TARGETS SUMMARY
10:08:39 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10:08:39 DISABLED test targets:
10:08:39 compiler-api-javax_tools_0
10:08:39 runtime-api-java_util-regex_0
10:08:39
10:08:39 PASSED test targets:
10:08:39 compiler-api-signaturetest_0 - Test results: passed: 2
10:08:39 runtime-api-java_lang-instrument_0 - Test results: passed: 10
10:08:39 runtime-api-java_lang-invoke_0 - Test results: passed: 110
10:08:39 runtime-api-java_lang-reflect_0 - Test results: passed: 143
10:08:39 runtime-api-java_lang-Package_0 - Test results: passed: 12
10:08:39 runtime-api-java_lang-String_0 - Test results: passed: 64
10:08:39 runtime-api-java_lang-StringBuilder_0 - Test results: passed: 25
10:08:39 runtime-api-java_lang-StringBuffer_0 - Test results: passed: 28
10:08:39 runtime-api-java_lang-ClassLoader_0 - Test results: passed: 35
10:08:39 runtime-api-java_lang-ModuleLayer_0 - Test results: passed: 23
10:08:39 runtime-api-java_lang-module_0 - Test results: passed: 163
10:08:39 runtime-api-java_lang-StackWalker_0 - Test results: passed: 8
10:08:39 runtime-api-java_lang-Module_class_0 - Test results: passed: 42
10:08:39 runtime-api-java_lang-SecurityManager_0 - Test results: passed: 2
10:08:39 runtime-api-java_lang-System_0 - Test results: passed: 26
10:08:39 runtime-api-java_lang-Class_0 - Test results: passed: 83
10:08:39 runtime-api-signaturetest_0 - Test results: passed: 20
10:08:39 runtime-api-modulegraph_0 - Test results: passed: 1
10:08:39 runtime-api-javax_naming-spi_0 - Test results: passed: 129
10:08:39
10:08:39 TOTAL: 21 EXECUTED: 19 PASSED: 19 FAILED: 0 DISABLED: 2 SKIPPED: 0
10:08:39 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10:08:39
10:08:39 TESTCASES RESULTS SUMMARY: passed: 32,585; failed: 1; error: 0; skipped: 0
10:08:39 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
10:08:39 ALL TESTS PASSED
...
Finished: SUCCESS
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the mismatch between the failed runtime-api-java_security_0 result and the TEST TARGETS SUMMARY, then trace the reporting path that produces TESTCASES RESULTS SUMMARY and the final build status. Done means failed targets appear in the summary and a failed test cannot produce ALL TESTS PASSED or Finished: SUCCESS.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.