adoptium / adoptium/aqa-tests

Improve openjdk exclusion file handling for new versions

Open
#5,932 2 comments 0 reactions 1 assignee Claimed by @sophia-guo View on GitHub
Dominant language
HTML
Stars
155
Forks
346
Avg merge
2d 9h
Merged PRs (30d)
36

Description

Currently, for new jdk version, a new exclusion file ProblemList_**openjdkversion**.txt is required to be added or the default one ( ProblemList_openjdk11.txt) will be used to avoid breaking the test job. Normally we will copy the most recent previous version (e.g, if the newest is 25, we copy the version 24) and add it as the newest one.

The may cause inaccurate exclusion. The old exclusion file may contain test cases that have been fixed or even enhanced in the latest version. Keeping them excluded unnecessarily reduces test coverage and prevent immediate detection of new failures. (Consider the openjdk developing process, backport always takes time or won't happen at all).

Instead of requiring a manually copied exclusion file for each new version we could modify the process to run without local exclusion file to allow all test cases to execute ( sync with [upstream openjdk](https://github.com/openjdk/jdk) ) and generate the exclusion file based on actual triage failures. Note generating a new exclusion file may take extra time if many tests fail.

This can be limited to impl=hotspot as openj9 may prefer different option https://github.com/adoptium/aqa-tests/issues/5890

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.