adoptium / adoptium/TKG

extended.perf Tests Executed in Incorrect Order

Open
#181 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
22
Forks
114
Avg merge
4d 2h
Merged PRs (30d)
8

Description

**Summary**

In one extended.perf run, each failed test target would be followed by the re-execution of earlier test targets.

Note that while stack overflows have been seen elsewhere, I've only seen this problem once.

**Details**

In a full extended.perf run, I expect each test target to be executed once apiece, in order. Here, though, we'd execute a test target, sometimes it would fail, and failed targets would be declared to be a different (earlier) test target, followed by the targets that were meant to be run after the different (incorrect) test target.

It's as if our position in the list of test targets is being repeatedly reset to the position of the earlier test target.

See [this issue](https://github.com/AdoptOpenJDK/openjdk-tests/issues/2514) for the full story.

This issue is raised specifically to cover the problem where the ordering of the test targets could be changed during execution. Raising this as a TKG issue as that's where the playlist file is passed in, so that's where I assume the list (and ordering) of test targets is meant to be stored.

**Link**
https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.perf_ppc64_aix/15/

**Grinder Link**
Attempting to reproduce. [Link.](https://ci.adoptopenjdk.net/job/Grinder/102/)
Result: Saw one StackOverflow, but we need more than one to reproduce this.

Bigger repro: https://ci.adoptopenjdk.net/job/Grinder/114/

**Repro Link**
https://ci.adoptopenjdk.net/job/Grinder/parambuild/?SDK_RESOURCE=upstream&TARGET=extended.perf&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=0&VENDOR_TEST_DIRS=&EXTRA_DOCKER_ARGS=&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&TEST_PARALLELIZATION_PARAMS=&PLATFORM=ppc64_aix&KEEP_REPORTDIR=false&PERSONAL_BUILD=false&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2FAdoptOpenJDK%2Fopenjdk-tests.git&LABEL=&TEST_OPTIONS_PARAMS=&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&NON_AQA_TEST_REPOS_HELP_TEXT=&ADOPTOPENJDK_BRANCH=master&LIGHT_WEIGHT_CHECKOUT=false&NON_AQA_TEST_REPOS=&ARTIFACTORY_SERVER=&TEST_REPO_PARAMS=&TEST_SELECTION_PARAMS=&TEST_PARALLELIZATION_PARAMS_HELP_TEXT=&JDK_SELECTION_PARAMS=&KEEP_WORKSPACE=false&USER_CREDENTIALS_ID=&JDK_VERSION=11&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=https%3A%2F%2Fgithub.com%2Fibmruntimes%2Fopenj9-openjdk-jdk11&PLATFORM_AND_MACHINE_HELP_TEXT=&RELEASE_TAG=&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&UPSTREAM_JOB_NAME=build-scripts%2Fjobs%2Fjdk11u%2Fjdk11u-aix-ppc64-openj9&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&PLATFORM_AND_MACHINE=&CUSTOM_TARGET=&VENDOR_TEST_SHAS=&JDK_BRANCH=openj9&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&POST_RUN_PARAMS_HELP_TEXT=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=&JDK_SELECTION_PARAMS_HELP_TEXT=&JDK_IMPL=openj9&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&TKG_SHA=&TEST_SELECTION_PARAMS_HELP_TEXT=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&OPENJDK_SHA=&NUM_MACHINES=&BUILD_LIST=perf&UPSTREAM_JOB_NUMBER=964&TEST_REPO_PARAMS_HELP_TEXT=&POST_RUN_PARAMS=&TIME_LIMIT=10&TEST_OPTIONS_PARAMS_HELP_TEXT=&JVM_OPTIONS=&PARALLEL=None

Contributor guide

Open the contributing guide

Research direction

Start by tracing TKG's handling of the playlist file passed for a full extended.perf run, then compare the Grinder 114 reproduction with the linked job output. Done means each test target executes once in playlist order, without failed targets resetting the position to an earlier target.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.