adoptium / adoptium/aqa-tests

trapme: tests failing when a testimage is not specified on a `CUSTOMIZED_SDK_URL`

Open
#5,179 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
155
Forks
346
Avg merge
2d 9h
Merged PRs (30d)
36

Description

I've been re-running the tests suites manually on some custom builds in order to test some modified compiler options on Linux as per https://github.com/adoptium/temurin-build/issues/3685

By default if you just specify the JDK tarball a number of the tests in the sanity suite will fail - for example this was the output from [sanity.openjk JDk22 without the testimage specified](https://ci.adoptium.net/job/Test_openjdk22_hs_sanity.openjdk_x86-64_linux/36/console):
```
18:39:09 jdk_lang_0 - Test results: passed: 999; failed: 4; error: 7
18:39:09 jdk_lang_1 - Test results: passed: 999; failed: 4; error: 7
18:39:09 jdk_lang_native_0 - Test results: error: 1
18:39:09 jdk_lang_native_1 - Test results: error: 1
18:39:09 jdk_foreign_0 - Test results: passed: 56; failed: 40; error: 4
18:39:09 jdk_foreign_1 - Test results: passed: 56; failed: 40; error: 4
```
(Note, I'm not 100% certain all of these are caused by the missing test image but many of them are, and a subsequent run with the fix had all of these passing)

Since this is "non-obvious" when you're just trying to run a test suite I think it would be useful to trap this situation or otherwise warn the user if a testimage has not been specified and the tests may require it. The above list may be adequate for determining which suites require it. Presumably these would also be a list of "expected failures" when using someone else's third party tarball which does not include a testimage download too and therefore would not be able to pass the tests.

The description of `CUSTOMIZED_SDK_URL` is also `Customized SDK url, need to set when SDK_RESOURCE=customized` and makes no mention of the special format to specify the testimage tarball (specify it after the main jdk tarball with a space character separating them) so it would be good to update the description to make that clear. I think chaging the behavior here would be a great usability improvement to stop people tripping up and having unexpected test case failures

Contributor guide

Open the contributing guide

Research direction

Start by tracing how CUSTOMIZED_SDK_URL and SDK_RESOURCE=customized are handled when launching the sanity suites, especially jdk_lang, jdk_lang_native, and jdk_foreign. Check how a missing testimage is detected and how the CUSTOMIZED_SDK_URL description is presented. Done means users receive a clear warning or trap and the required testimage format is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.