adoptium / adoptium/aqa-test-tools
Perf Compare (TRSS) regression percentage calculation for startup and footprint benchmarks
- Dominant language
- Jupyter Notebook
- Stars
- 33
- Forks
- 97
- Avg merge
- 7h 9m
- Merged PRs (30d)
- 5
Description
We see that currently on Perf Compare (TRSS) the regression percentage for SUFP is being calculated by dividing the baseline result by the test result (above 100% is an improvement). In the WAS Performance team we have been doing it the other way around, test result / baseline result (below 100% is an improvement). We also add a +/- 3% threshold to account for test variability.
```
const curDiff = curHigherBetter ? curTestScore / curBaselineScore : curBaselineScore / curTestScore;
// Row colours based on improvement or regression
if (curDiff > 1) {
curColor = 'improvement';
} else if (curDiff < 1) {
curColor = 'regression';
} else {
curColor = 'nochange';
}
```
```
const curDiff = curTestScore / curBaselineScore;
// WAS Performance Threshold
const regressionThreshold = 0.03;
// Row colours based on improvement or regression
if ( (curHigherBetter && curDiff > 1 + regressionThreshold) || (!curHigherBetter && curDiff < 1 - regressionThreshold) ) {
curColor = 'improvement';
} else if ( (curHigherBetter && curDiff < 1 - regressionThreshold) || (!curHigherBetter && curDiff > 1 + regressionThreshold) ) {
curColor = 'regression';
} else {
curColor = 'nochange';
}
```
Contributor guide
Research direction
Start at the Perf Compare (TRSS) regression percentage calculation for SUFP, using the current comparison logic and the WAS Performance examples in the issue as the entry point. Verify startup and footprint benchmark row colours against the proposed ratio and ±3% threshold; done means improvements, regressions, and no-change results are classified as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100