adoptium / adoptium/aqa-test-tools

Incorrect Test Targets Displayed for single extended.perf run

Open
#407 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
33
Forks
97
Avg merge
7h 9m
Merged PRs (30d)
5

Description

**Summary**

In a deeply unusual extended.perf run, several failed test targets are identified by TRSS as "disabled", and several more are identified as "failed" when they actually passed.

https://trss.adoptopenjdk.net/allTestsInfo?buildId=60735c4947db975731f280c2&limit=5&hasChildren=false

**Details**

This job is unusual in several ways, as described [here](https://github.com/AdoptOpenJDK/openjdk-tests/issues/2514).

In short, every time a test target fails:
- The message at the end of the test says, incorrectly, that it's a different, earlier target that failed.
- The progression through the list of test targets is somehow reset to that of the earlier test target.
- The next targets to be run will be the ones that are the correct successors to the earlier target.

As a result, dismissing this bug as GIGO would be fair, though perhaps TRSS could be hardened to these sorts of shenanigans?

Also note that this issue has only been seen once so far, across all versions, platforms, and VMs (OpenJ9). See the repro link below for an attempt to reproduce.

**Repro Links**
https://ci.adoptopenjdk.net/job/Grinder/102/
Result: Saw one StackOverflow, but we need more than one to reproduce this.

Bigger repro: https://ci.adoptopenjdk.net/job/Grinder/114/

**Grinder Link**
https://ci.adoptopenjdk.net/job/Grinder/parambuild/?SDK_RESOURCE=upstream&TARGET=extended.perf&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=0&VENDOR_TEST_DIRS=&EXTRA_DOCKER_ARGS=&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&TEST_PARALLELIZATION_PARAMS=&PLATFORM=ppc64_aix&KEEP_REPORTDIR=false&PERSONAL_BUILD=false&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2FAdoptOpenJDK%2Fopenjdk-tests.git&LABEL=&TEST_OPTIONS_PARAMS=&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&NON_AQA_TEST_REPOS_HELP_TEXT=&ADOPTOPENJDK_BRANCH=master&LIGHT_WEIGHT_CHECKOUT=false&NON_AQA_TEST_REPOS=&ARTIFACTORY_SERVER=&TEST_REPO_PARAMS=&TEST_SELECTION_PARAMS=&TEST_PARALLELIZATION_PARAMS_HELP_TEXT=&JDK_SELECTION_PARAMS=&KEEP_WORKSPACE=false&USER_CREDENTIALS_ID=&JDK_VERSION=11&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=https%3A%2F%2Fgithub.com%2Fibmruntimes%2Fopenj9-openjdk-jdk11&PLATFORM_AND_MACHINE_HELP_TEXT=&RELEASE_TAG=&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&UPSTREAM_JOB_NAME=build-scripts%2Fjobs%2Fjdk11u%2Fjdk11u-aix-ppc64-openj9&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&PLATFORM_AND_MACHINE=&CUSTOM_TARGET=&VENDOR_TEST_SHAS=&JDK_BRANCH=openj9&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&POST_RUN_PARAMS_HELP_TEXT=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=&JDK_SELECTION_PARAMS_HELP_TEXT=&JDK_IMPL=openj9&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&TKG_SHA=&TEST_SELECTION_PARAMS_HELP_TEXT=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&OPENJDK_SHA=&NUM_MACHINES=&BUILD_LIST=perf&UPSTREAM_JOB_NUMBER=964&TEST_REPO_PARAMS_HELP_TEXT=&POST_RUN_PARAMS=&TIME_LIMIT=10&TEST_OPTIONS_PARAMS_HELP_TEXT=&JVM_OPTIONS=&PARALLEL=None

Contributor guide

Open the contributing guide

Research direction

Start with the TRSS allTestsInfo result for build 60735c4947db975731f280c2, then compare it with the Grinder 102 and 114 reproduction links. Trace how failed and passed targets are classified and how target progression is reset, using the reported behavior as the completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.