Implement scripts for detection of performance regressions across CUB versions
Open
cub
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 486
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 295
Description
Tuning infrastructure is currently used to find better tuning variants. Reusing it for the detection of performance regressions is preferable since there should be a single source of truth for workloads and benchmarks we care about. The detection of performance regressions should be based on sample distribution analysis. We can start with Mann–Whitney U test, and if it's likely that two distributions do not belong to a single one, proceed with the ratio function based on the Harrell-Davis quantile estimator. It might also be worth it to experiment with outlier detection.
Contributor guide
Assessment
This issue has not been assessed yet.