adoptium / adoptium/aqa-test-tools

Review "Pass %" metric and consider a new "Pipeline %" metric

Open
#752 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
33
Forks
97
Avg merge
7h 9m
Merged PRs (30d)
5

Description

Pass percentage is calculated by Passed/Executed and does not currently take into account aborted tests. While we originally wanted **Test** Results Summary Service (TRSS) to summarize the results of tests that actually ran, it is likely more useful to include info about Aborted tests (which is often an indication of Infra issues).

For example, in the past [weekend pipeline runs](https://trss.adoptium.net/resultSummary?parentId=638b8375ed0b30773b403e9d), most test jobs had to abort because there were no machines found online, so 7 of the 11 platforms could not launch AQA tests, due to
```
[Pipeline] nodesByLabel
13:19:17 Could not find any nodes with 'ci.role.test&&hw.arch.x86&&sw.os.windows&&sw.tool.docker' label
[Pipeline] timeout
13:19:17 Timeout expired 0 ms ago
13:19:17 Cancelling nested steps due to timeout
...
Finished: ABORTED
```

Screen Shot 2022-12-05 at 10 40 38 AM

The pass % shows to be 96.78% which are of the tests that did launch and could be run. The overall health of the weekly pipeline was poor, so it may be good to either change how Pass% is calculated, or figure out a calculation for a new metric, something called Pipeline% which includes "Aborted" jobs in the final metric. Pipeline % could measure the number of jobs in the pipeline that could run over the number of jobs that were launched.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing how TRSS currently calculates Pass% from Passed and Executed results, then examine how aborted jobs appear in the weekly pipeline summary. Done means agreeing whether Pass% should change or defining a separate Pipeline% metric, including its inputs and expected interpretation for aborted jobs.

Written by the indexing model from the issue text.

Assessment

Domain
analytics, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.