cloud-bulldozer / cloud-bulldozer/benchmark-comparison

account for variance in samples

Open
#10 17 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
23
PR merge metrics
No merged PRs in 30d

Description

Current implementation of touchstone calculates averages and then compares them. This approach does not take into account variation in samples for baseline, and variation in new SUT. So you cannot tell if the change in average is statistically significant. There are established statistical methods for incorporating variance into the comparison, as described here:

https://mojo.redhat.com/docs/DOC-1089994

which is basically describing how to use [scipy.ttest_ind() ](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html)function. It would be also good to monitor the % deviation of the baseline and new-run samples to see whether we *can* determine whether a regression has occurred or not. This kind of analysis can prevent false positives and negatives and avoid wasting time on unnecessary investigations.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the touchstone implementation that currently calculates and compares averages, then review scipy.ttest_ind() and the linked statistical guidance. Define how significance and percentage deviation should be reported for baseline and new-run samples; the work is done when comparisons account for sample variance and reduce false regression results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.