cloud-bulldozer / cloud-bulldozer/benchmark-comparison
account for variance in samples
- Dominant language
- Python
- Stars
- 1
- Forks
- 23
- PR merge metrics
- No merged PRs in 30d
Description
Current implementation of touchstone calculates averages and then compares them. This approach does not take into account variation in samples for baseline, and variation in new SUT. So you cannot tell if the change in average is statistically significant. There are established statistical methods for incorporating variance into the comparison, as described here:
https://mojo.redhat.com/docs/DOC-1089994
which is basically describing how to use [scipy.ttest_ind() ](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html)function. It would be also good to monitor the % deviation of the baseline and new-run samples to see whether we *can* determine whether a regression has occurred or not. This kind of analysis can prevent false positives and negatives and avoid wasting time on unnecessary investigations.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the touchstone implementation that currently calculates and compares averages, then review scipy.ttest_ind() and the linked statistical guidance. Define how significance and percentage deviation should be reported for baseline and new-run samples; the work is done when comparisons account for sample variance and reduce false regression results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100