adoptium / adoptium/aqa-test-tools

Perf Compare (TRSS) regression percentage calculation for startup and footprint benchmarks

Open
#297 1 comment 0 reactions 0 assignees View on GitHub
discussion
Dominant language
Jupyter Notebook
Stars
33
Forks
97
Avg merge
7h 9m
Merged PRs (30d)
5

Description

We see that currently on Perf Compare (TRSS) the regression percentage for SUFP is being calculated by dividing the baseline result by the test result (above 100% is an improvement). In the WAS Performance team we have been doing it the other way around, test result / baseline result (below 100% is an improvement). We also add a +/- 3% threshold to account for test variability.

```
const curDiff = curHigherBetter ? curTestScore / curBaselineScore : curBaselineScore / curTestScore;
// Row colours based on improvement or regression
if (curDiff > 1) {
curColor = 'improvement';
} else if (curDiff < 1) {
curColor = 'regression';
} else {
curColor = 'nochange';
}
```

```
const curDiff = curTestScore / curBaselineScore;
// WAS Performance Threshold
const regressionThreshold = 0.03;
// Row colours based on improvement or regression
if ( (curHigherBetter && curDiff > 1 + regressionThreshold) || (!curHigherBetter && curDiff < 1 - regressionThreshold) ) {
curColor = 'improvement';
} else if ( (curHigherBetter && curDiff < 1 - regressionThreshold) || (!curHigherBetter && curDiff > 1 + regressionThreshold) ) {
curColor = 'regression';
} else {
curColor = 'nochange';
}
```

Contributor guide

Open the contributing guide

Research direction

Start at the Perf Compare (TRSS) regression percentage calculation for SUFP, using the current comparison logic and the WAS Performance examples in the issue as the entry point. Verify startup and footprint benchmark row colours against the proposed ratio and ±3% threshold; done means improvements, regressions, and no-change results are classified as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.