adoptium / adoptium/aqa-tests

EPIC: Enable a Disabled tests pipeline or workflow

Open
#1,853 11 comments 0 reactions 1 assignee Claimed by @adamfarley View on GitHub
canosp enhancement epic
Dominant language
HTML
Stars
155
Forks
346
Avg merge
2d 9h
Merged PRs (30d)
36

Description

**Goal:**

To create a pipeline or workflow that evaluates disabled tests for future re-enabling.

**Plan Summary:**

- [x] Step 1: To create a parser that reads the ProblemLists and Playlists. [exclude_parser.py](https://github.com/adoptium/aqa-tests/blob/master/scripts/disabled_tests/exclude_parser.py)
- [x] Step 2: To create a tool that checks if an exclusion's issue is resolved. https://github.com/adoptium/aqa-tests/issues/2701
- [x] Step 3: To create a github workflow that can use both 1 and 2 when committing an aqa-test PR.
- [x] Step 3.1: Create workflow. https://github.com/adoptium/aqa-tests/issues/3917
- [x] Step 3.2: Generate json of disabled tests. https://github.com/adoptium/aqa-tests/issues/3345
- [x] Step 4: To create a tool for running 1 and 2 easily in a user's sandbox. https://github.com/adoptium/aqa-tests/pull/6993
- [x] Step 5: Repair and upgrade the parser: https://github.com/adoptium/aqa-tests/issues/6994
- [x] Step 5.1: Fix the parse-issues.yml workflow so that errors cause it to fail
- [x] Step 5.2: Resolve all errors that cause it to fail.
- [x] Step 5.3: Teach it the difference between fixed and "will not fix".
- Note: May need a new aqa-tests label to indicate this (if there isn't one already).
- [x] Step 5.4: Figure out if the fix for each closed issue is in the non-dev src.
- [x] Step 5.5: Add an argument that simplifies the output.
- [ ] Step 6: Have the bash script produce grinder links for test runs. https://github.com/adoptium/aqa-tests/issues/7204
- [x] 6.1: Parse parse_issues.py output. - 0.5pd
- [ ] 6.2: Generate grinders for the "fixed" tests. - 1.5pd
- [ ] 6.3: Collect the results and identify passes. - 1.5pd
- [ ] 6.4: Create a PR, or generate output that can be used to create a PR. - 1pd
- [ ] 6.5: Post results to Slack. 0.5pd
- Slack message summary should be simple. One line per version.
- E.g. JDKxx: 100 excludes, 25 issues closed, 18 tests safe to unexclude.
- [ ] 6.6: Schedule job to run fortnightly. 0.5pd

**Notes**

This tool/pipeline/workflow could:
- Check the state of the issue that the test was disabled under
- Run all disabled test targets whose issues are resolved
- Present a report after checking the results, which says if any really are in a state to reinclude/reenable.

Since there are various granularities on how we can and do disable tests, this work would look at several parts:

- disabled test targets in playlists
- disabled openjdk tests in openjdk subdirectory (in problemlist*.txt files)
- disabled functional tests in openj9 repo (several types of exclusions, testNG, cmdlinetester, others?) - optional for 1st version of this feature
- https://github.com/adoptium/aqa-tests/issues/3917 (check when the playlists are changed and report if there are any exclusions that should be assessed, or any parsing issues with the playlist)

Update from [comment below](https://github.com/adoptium/aqa-tests/issues/1853#issuecomment-871612843) where there are several subtasks as part of this EPIC:
- #2698
- #2699
- #2701
- #2702

We have many pieces of this EPIC completed, and now it is time to integrate the pieces into a single working pipeline.

**Summary of progress**

- We currently have a script that parses the exclude files and creates a json that can be used to run only the excluded tests:
- https://github.com/adoptium/aqa-tests/blob/master/scripts/disabled_tests/exclude_parser.py
- We also have a groovy script that uses the parser to run disabled tests:
- https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/triggerGrinder.groovy
- Lastly, we have a jenkins job that uses that groovy script:
- https://ci.adoptium.net/view/Test_grinder/job/AutoReenableJob

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.