adoptium / adoptium/aqa-tests

Capture TKG test result logic may not work with ITERATIONS

Open
#4,092 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
155
Forks
346
Avg merge
2d 9h
Merged PRs (30d)
36

Description

**Describe the bug**
With the capture TKG test result, we can visualize the test summary in the Jenkins job (see below). This is really handy for the users.


![image](https://user-images.githubusercontent.com/19273206/198327679-6319182b-fa84-4533-80c7-0ebbad884137.png)

However, this feature may give false positives or false negatives when running with ITERATIONS. In the following case ([Grinder link](https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/29046/)), the test has an intermittent failure. It passed 3/5. `TEST TARGETS SUMMARY` shows a false positive.

![image](https://user-images.githubusercontent.com/19273206/198328107-58235ca3-6441-40e0-abe3-61ff0aad1364.png)

I think we should update the logic to capture all iteration results.
https://github.com/adoptium/aqa-tests/blob/9166b61bddc5156f6d3182ff536a5e035293452a/buildenv/jenkins/JenkinsfileBase#L673-L710

As we enable [Rerun failed tests automatically feature](https://github.com/adoptium/aqa-tests/issues/3431), this issue will show up in the nightly runs with rerun enabled.
Also, the good news is that the job status is correct (i.e., UNSTABLE) in this case because we also check TAP files.

FYI @renfeiw

Contributor guide

Open the contributing guide

Research direction

Read buildenv/jenkins/JenkinsfileBase around lines 673-710 and inspect the Grinder job linked in the report, focusing on how ITERATIONS are represented in the captured result. Compare the summary with the 3/5 outcome and update the logic so all iteration results are reflected without changing the correct UNSTABLE job status from TAP files.

Written by the indexing model from the issue text.

Assessment

Domain
ci-cd, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.