adoptium / adoptium/aqa-systemtest

Adding ability to run multiple subsets of JCK tests at once

Open
#37 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
27
Forks
70
PR merge metrics
No merged PRs in 30d

Description

For the convenience of executing JCK tests, there is a need to run arbitrary subsets of JCK tests (e.g. `api/java_lang` and `api/java_math/BigDecimal`).
Based on the description in `openjdk.test.jck/include/test_targets.mk`, there is a `test.jck.custom` target which can help developer to execute arbitrary subset of JCK tests. But it seems this target can only accept one test subset at once, if multiple subsets were provided, it will throw exception.

For example:
1. cmd like this `make -k test.jck.custom JCKVERSION=jck9 JCKTESTSUITE=RUNTIME JCKTEST="api/java_lang"` will successfully generate the STF JCK test script and start running the api/java_lang JCK test.

2. cmd like this `make -k test.jck.custom JCKVERSION=jck9 JCKTESTSUITE=RUNTIME JCKTEST="api/java_lang api/java_math/BigDecimal"` will failed the STF JCK test execution with error message

```STF 18:06:36.920 - Running command: /java9/bin/java -jar /root/systemtest_prereqs/jck/jck9/JCK-runtime-9/lib/jtlite.jar -config /root/git/openjdk-systemtest/openjdk.test.jck/config/jck9/runtime.jti @/tmp/stf/20171109-180633-Jck/results/JTB/stf_generated.jtb
STF 18:06:36.920 - Redirecting stderr to /tmp/stf/20171109-180633-Jck/results/3.JCK.stderr
STF 18:06:36.920 - Redirecting stdout to /tmp/stf/20171109-180633-Jck/results/3.JCK.stdout
STF 18:06:36.933 - Monitoring processes: JCK
JCK stderr + set jck.env.runtime.testExecute.otherOpts " --add-modules java.xml.ws.annotation,java.xml.bind,java.xml.ws,java.activation,java.corba -Xdump:system:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -Xdump:snap:none -Xdump:snap:events=gpf+abort+traceassert+corruptcache -Xdump:java:none -Xdump:java:events=gpf+abort+traceassert+corruptcache -Xdump:heap:none -Xdump:heap:events=gpf+abort+traceassert+corruptcache "
JCK stderr Configuration file can not be imported due to question with specified tag "jck.env.runtime.testExecute.otherOpts" was not found on path. The path is:jck.intro
JCK stderr jck.env.simpleOrAdvanced (advanced)
JCK stderr jck.env.envName (jck_runtime)
JCK stderr jck.env.description (JCK9 runtime template jti for ST...)
JCK stderr jck.env.platform (jre)
JCK stderr jck.tests.chooseTests (Yes)
JCK stderr jck.tests.treeOrFile (tree)
JCK stderr jck.tests.tests (api/java_lang api/java_math/BigD...)
JCK stderr jck.env.testPlatform.intro
JCK stderr jck.env.testPlatform.nativeCode (Yes)
JCK stderr jck.env.testPlatform.multiJVM (Yes)
JCK stderr jck.env.testPlatform.useAgent (No)
JCK stderr jck.env.testPlatform.os (Current system)
JCK stderr jck.env.runtime.intro
JCK stderr jck.env.runtime.otherJVM
JCK stderr jck.env.runtime.testExecute.intro
JCK stderr jck.env.runtime.testExecute.cmdAsFile (/java9/bin/java)
JCK stderr jck.env.runtime.testExecute.classpath (command line option)
JCK stderr jck.env.runtime.testExecute.classpathOpt (-classpath #)
JCK stderr jck.env.runtime.testExecute.additionalClasspath ()
JCK stderr jck.env.moduleSystem.vmOptions (vmAddModsOptionTemplate=--add-mo...)
JCK stderr jck.env.runtime.testExecute.nativeLibsLinkage (dynamic)
JCK stderr jck.env.runtime.testExecute.libPath (environment variable)
JCK stderr jck.env.runtime.testExecute.libPathEnv (will_be_set_by_test_automation_a...)
JCK stderr jck.env.runtime.testExecute.nativeLibrariesLocation (Yes)
JCK stderr jck.env.runtime.testExecute.nativeLibPathFileValue (will_be_set_by_test_automation_a...)
JCK stderr jck.env.runtime.testExecute.cannot_read
JCK stderr
STF 18:06:38.599 - **FAILED** Process JCK ended with exit code (3) and not the expected exit code/s (0)
STF 18:06:38.599 - Monitoring Report Summary:
STF 18:06:38.599 - o Process JCK ended with exit code (3) and not the expected exit code/s (0)
STF 18:06:38.600 - Killing processes: JCK
STF 18:06:38.600 - o Process JCK is not running
**FAILED** at step 3 (Running JCK in multijvm way with Agent off). Expected return value=0 Actual=1 at /tmp/stf/20171109-180633-Jck/execute.pl line 93.
STF 18:06:38.619 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 18:06:38.620 -
STF 18:06:38.620 - ==================== T E A R D O W N ====================
```

Although we can use a loop (in script) to run a list of subsets, the test summary outputs will be separated and hard to track which tests passed and which tests failed. I think we need the STF JCK test to be able to run multiple subsets of JCK test at once, as we want the final test summary to be combined together.

Contributor guide

Open the contributing guide

Research direction

Start with openjdk.test.jck/include/test_targets.mk and the test.jck.custom target, then trace how the JCKTEST value is passed into the generated STF JCK test script. Reproduce the single-subset and multiple-subset commands from the issue; done means multiple subsets execute in one STF run and produce a combined test summary.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.